[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:56 dist-test-slave-3h01 krb5kdc[18235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:56 dist-test-slave-3h01 krb5kdc[18235](info): set up 2 sockets
Sep 29 16:34:56 dist-test-slave-3h01 krb5kdc[18235](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:34:56.945178 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:34:57 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:34:57.116220 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33037
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:57.423194 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.423519 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.423621 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.430851 18248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:57.430971 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.431018 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.431079 18248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:57.431121 18248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:57.441038 18248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:57.442433 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:57.443818 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:34:57.456238 18248 server_base.cc:1034] running on GCE node
W20260929 16:34:57.456141 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.456117 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.456084 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:57.457392 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:57.458880 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:57.460105 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790699697460007 us; error 72 us; skew 500 ppm
I20260929 16:34:57.460448 18248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:57 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:57.468320 18248 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:34:57.470002 18248 webserver.cc:460] Webserver started at http://127.17.205.62:43421/ using document root <none> and password file <none>
I20260929 16:34:57.470745 18248 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:57.470862 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:57.471161 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:57.474160 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "a4805b116dae457eb3e690cb508e4aa8"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-3h01"
server_key: "8f7a0c9a5918f03373980260bfdd2ece"
server_key_iv: "f9d86db12955eb5f13e8c83bb8939c5d"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.474931 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "a4805b116dae457eb3e690cb508e4aa8"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-3h01"
server_key: "8f7a0c9a5918f03373980260bfdd2ece"
server_key_iv: "f9d86db12955eb5f13e8c83bb8939c5d"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.480226 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:57.483716 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.485092 18248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:34:57.485251 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a4805b116dae457eb3e690cb508e4aa8"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-3h01"
server_key: "8f7a0c9a5918f03373980260bfdd2ece"
server_key_iv: "f9d86db12955eb5f13e8c83bb8939c5d"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.485427 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:57.509821 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:57.513834 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:57.514097 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:57.530841 18248 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:33037
I20260929 16:34:57.530862 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:33037 every 8 connection(s)
I20260929 16:34:57.532220 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:57.535841 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:57.541277 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18248
I20260929 16:34:57.541576 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:34:57.543376 18228 external_mini_cluster.cc:1428] Setting key a55026b07332da1959b2284a95f704e4
I20260929 16:34:57.545674 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8: Bootstrap starting.
I20260929 16:34:57.549551 18315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:57.550474 18315 log.cc:824] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:57.553030 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8: No bootstrap required, opened a new log
I20260929 16:34:57.561188 18315 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } }
I20260929 16:34:57.561570 18315 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:57.561678 18315 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4805b116dae457eb3e690cb508e4aa8, State: Initialized, Role: FOLLOWER
I20260929 16:34:57.562242 18315 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } }
I20260929 16:34:57.562450 18315 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:57.562567 18315 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:57.562738 18315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:57.564878 18315 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } }
I20260929 16:34:57.565351 18315 leader_election.cc:302] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4805b116dae457eb3e690cb508e4aa8; no voters:
I20260929 16:34:57.565948 18315 leader_election.cc:288] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:57.566048 18320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:57.567072 18320 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [term 1 LEADER]: Becoming Leader. State: Replica: a4805b116dae457eb3e690cb508e4aa8, State: Running, Role: LEADER
I20260929 16:34:57.567721 18320 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } }
I20260929 16:34:57.568228 18315 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 16:34:57 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:34:57.570538 18321 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4805b116dae457eb3e690cb508e4aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } } }
I20260929 16:34:57.570912 18321 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:57.571427 18322 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4805b116dae457eb3e690cb508e4aa8. Latest consensus state: current_term: 1 leader_uuid: "a4805b116dae457eb3e690cb508e4aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4805b116dae457eb3e690cb508e4aa8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33037 } } }
I20260929 16:34:57.571743 18322 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:57.573207 18325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:57.578581 18325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:57.581851 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.546998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.17.205.62:33037)
0929 16:34:57.547803 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.547818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.548972 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 16:34:57.557451 (+ 8479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.557473 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.557894 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.558271 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:57.561168 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.562688 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.564373 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.565106 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.565310 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.571639 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:57.571655 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:57.571680 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:57.571733 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:57.575966 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.577126 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.577136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.577145 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.577377 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.577999 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.578005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.578009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.578805 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:57.579939 (+ 1134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.580235 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.580667 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":389,"threads_started":1}
I20260929 16:34:57.588970 18325 catalog_manager.cc:1347] Generated new cluster ID: 696d0390e3014e188f3476debf3a544b
I20260929 16:34:57.589076 18325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:57.602999 18325 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:57.604894 18325 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:57.619660 18325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4805b116dae457eb3e690cb508e4aa8: Generated new TSK 0
I20260929 16:34:57.620558 18325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:34:57.662164 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:57.980046 18341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.980329 18341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.980458 18341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.987522 18341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.987650 18341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.987811 18341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:34:57.997545 18341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:57.998811 18341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.000181 18341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.012566 18347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.012585 18349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.012621 18346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.013428 18341 server_base.cc:1034] running on GCE node
I20260929 16:34:58.014039 18341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.014948 18341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.016193 18341 hybrid_clock.cc:648] HybridClock initialized: now 1790699698016122 us; error 61 us; skew 500 ppm
I20260929 16:34:58.016534 18341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:58 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:58.019598 18341 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:34:58.021121 18341 webserver.cc:460] Webserver started at http://127.17.205.1:40445/ using document root <none> and password file <none>
I20260929 16:34:58.021786 18341 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.021883 18341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.022149 18341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.025003 18341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "560822781053471c81bff56c1d72dfc4"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "0a9d6a7a16120c71031329667327a693"
server_key_iv: "4f4f58f59c3ce96a6325cd8603217c93"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.025704 18341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "560822781053471c81bff56c1d72dfc4"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "0a9d6a7a16120c71031329667327a693"
server_key_iv: "4f4f58f59c3ce96a6325cd8603217c93"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.030771 18341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:34:58.034288 18356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.035712 18341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:58.035840 18341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "560822781053471c81bff56c1d72dfc4"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "0a9d6a7a16120c71031329667327a693"
server_key_iv: "4f4f58f59c3ce96a6325cd8603217c93"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.035998 18341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.050061 18341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.054332 18341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.054565 18341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.055481 18341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:58.056933 18341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:58.057016 18341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.057111 18341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:58.057154 18341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.086406 18341 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:36879
I20260929 16:34:58.086436 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:36879 every 8 connection(s)
I20260929 16:34:58.087745 18341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:58.091568 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18341
I20260929 16:34:58.091795 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:58.092420 18228 external_mini_cluster.cc:1428] Setting key 20b740503c38265b2939034c590d8cb9
Sep 29 16:34:58 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:34:58.108196 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.090319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:46755 (local address 127.17.205.62:33037)
0929 16:34:58.090727 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.090731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.091775 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 16:34:58.093697 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.093702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.093783 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.093873 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.095920 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.096642 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.098792 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.099399 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.099522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.102759 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.102764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.102768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:58.102802 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.105189 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:58.105884 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:58.105888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:58.105890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:58.106009 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:58.106413 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:58.106417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:58.106419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:58.106720 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.106848 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.107603 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.107832 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":214,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:34:58.109340 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.090792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33037 (local address 127.17.205.1:46755)
0929 16:34:58.091540 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.091584 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.093406 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.094134 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.094143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.094885 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:58.095734 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.095761 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.096798 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.096804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.098649 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.098664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.099598 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.099603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.099811 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.100442 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:58.100464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:58.102542 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:58.105328 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:58.105337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:58.105356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:58.105770 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:58.106118 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:58.106123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:58.106127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:58.106311 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:58.106836 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:58.106844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:58.107192 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.107507 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.108059 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":290,"threads_started":1}
I20260929 16:34:58.110816 18469 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33037
I20260929 16:34:58.111143 18469 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:58.111876 18469 heartbeater.cc:511] Master 127.17.205.62:33037 requested a full tablet report, sending...
I20260929 16:34:58.114112 18280 ts_manager.cc:194] Registered new tserver with Master: 560822781053471c81bff56c1d72dfc4 (127.17.205.1:36879)
I20260929 16:34:58.116065 18280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:46755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:34:58.130450 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:58.438261 18475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:58.438566 18475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:58.438685 18475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:58.445986 18475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:58.446127 18475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:58.446283 18475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:34:58.455960 18475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:58.457261 18475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.458590 18475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.471540 18483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.471664 18480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.471699 18481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.472179 18475 server_base.cc:1034] running on GCE node
I20260929 16:34:58.472787 18475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.473728 18475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.474921 18475 hybrid_clock.cc:648] HybridClock initialized: now 1790699698474865 us; error 53 us; skew 500 ppm
I20260929 16:34:58.475243 18475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:58 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:58.478149 18475 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:34:58.479557 18475 webserver.cc:460] Webserver started at http://127.17.205.2:43811/ using document root <none> and password file <none>
I20260929 16:34:58.480219 18475 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.480316 18475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.480567 18475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.483413 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "8d5e9ce2bcf44cc6aedcb5335f7e4d57"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "5154c6aaa1fafaf1146e80abe206b4ea"
server_key_iv: "5c7e416eba4d59af24288a8617bbf2ea"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.484153 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "8d5e9ce2bcf44cc6aedcb5335f7e4d57"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "5154c6aaa1fafaf1146e80abe206b4ea"
server_key_iv: "5c7e416eba4d59af24288a8617bbf2ea"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.489102 18475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:34:58.492481 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.493831 18475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:58.493983 18475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "8d5e9ce2bcf44cc6aedcb5335f7e4d57"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "5154c6aaa1fafaf1146e80abe206b4ea"
server_key_iv: "5c7e416eba4d59af24288a8617bbf2ea"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.494164 18475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.512672 18475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.516486 18475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.516734 18475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.517614 18475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:58.518860 18475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:58.518927 18475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.519016 18475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:58.519057 18475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.549595 18475 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:35907
I20260929 16:34:58.549638 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:35907 every 8 connection(s)
I20260929 16:34:58.551090 18475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:58.555188 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18475
I20260929 16:34:58.555397 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:58.555790 18228 external_mini_cluster.cc:1428] Setting key 7b7eec808bd0d0db3e44aa81c82c9ec0
Sep 29 16:34:58 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:34:58.569646 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.553512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:52455 (local address 127.17.205.62:33037)
0929 16:34:58.553673 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.553678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.554850 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 16:34:58.556710 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.556715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.556797 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.556883 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.558433 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.559114 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.560632 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.561020 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.561123 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.564351 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.564357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.564361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:58.564392 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.566716 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:58.567427 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:58.567431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:58.567433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:58.567563 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:58.568004 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:58.568008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:58.568010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:58.568265 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.568409 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.569106 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.569332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:34:58.570787 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.553998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33037 (local address 127.17.205.2:52455)
0929 16:34:58.554676 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.554717 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.556447 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.557124 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.557132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.557730 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:58.558278 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.558296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.559287 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.559291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.560501 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.560512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.561220 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.561226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.561444 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.562027 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:58.562052 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:58.564145 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:58.566842 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:58.566851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:58.566869 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:58.567314 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:58.567668 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:58.567673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:58.567676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:58.567903 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:58.568376 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:58.568384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:58.568772 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.569018 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.569570 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":249,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:34:58.572180 18603 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33037
I20260929 16:34:58.572516 18603 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:58.573359 18603 heartbeater.cc:511] Master 127.17.205.62:33037 requested a full tablet report, sending...
I20260929 16:34:58.574769 18280 ts_manager.cc:194] Registered new tserver with Master: 8d5e9ce2bcf44cc6aedcb5335f7e4d57 (127.17.205.2:35907)
I20260929 16:34:58.575573 18280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:52455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:34:58.590255 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:58.893867 18608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:58.894157 18608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:58.894276 18608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:58.901711 18608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:58.901835 18608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:58.901988 18608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:34:58.911826 18608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:58.913196 18608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.914512 18608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.927305 18613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.927268 18616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.927325 18614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.928655 18608 server_base.cc:1034] running on GCE node
I20260929 16:34:58.929266 18608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.930238 18608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.931429 18608 hybrid_clock.cc:648] HybridClock initialized: now 1790699698931374 us; error 49 us; skew 500 ppm
I20260929 16:34:58.931743 18608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:58 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:58.934816 18608 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:34:58.936379 18608 webserver.cc:460] Webserver started at http://127.17.205.3:44445/ using document root <none> and password file <none>
I20260929 16:34:58.937053 18608 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.937172 18608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.937456 18608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.940443 18608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "c0308af6584b4e1aa3409b5f4ce29051"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "be02a768a8fc6120fe1ffd468b20060a"
server_key_iv: "abaf80c95989b669b57dba60b2643350"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.941175 18608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "c0308af6584b4e1aa3409b5f4ce29051"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "be02a768a8fc6120fe1ffd468b20060a"
server_key_iv: "abaf80c95989b669b57dba60b2643350"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.946555 18608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:34:58.950271 18623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.951733 18608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:58.951891 18608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "c0308af6584b4e1aa3409b5f4ce29051"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-3h01"
server_key: "be02a768a8fc6120fe1ffd468b20060a"
server_key_iv: "abaf80c95989b669b57dba60b2643350"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.952077 18608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.996197 18608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:59.000836 18608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:59.001132 18608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:59.002101 18608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:59.003463 18608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:59.003553 18608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:59.003677 18608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:59.003736 18608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:59.031153 18608 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:34871
I20260929 16:34:59.031198 18735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:34871 every 8 connection(s)
I20260929 16:34:59.032460 18608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:59.035243 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18608
I20260929 16:34:59.035431 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:59.035856 18228 external_mini_cluster.cc:1428] Setting key 94288d4282d64b0ad435d76ca10a2c20
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:34:59.053843 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.035046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:54177 (local address 127.17.205.62:33037)
0929 16:34:59.035272 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.035279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.036758 (+ 1479us) server_negotiation.cc:407] Connection header received
0929 16:34:59.038732 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.038737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.038815 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.038910 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.040753 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.041773 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.043367 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.043957 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.044097 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.048271 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.048276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.048280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.048314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.050824 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.051646 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.051652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.051656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.051778 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.052180 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.052184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.052188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.052449 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.052580 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.053291 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.053514 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:59.054836 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.035733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33037 (local address 127.17.205.3:54177)
0929 16:34:59.036560 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:59.036612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:59.038471 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:59.039140 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:59.039160 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:59.039773 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:59.040546 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:59.040572 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.041933 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.041939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:59.043183 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:59.043198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.044226 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.044236 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:59.044536 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.045286 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:59.045314 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:59.048033 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:59.050939 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:59.050950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:59.050969 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:59.051532 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:59.051873 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:59.051879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:59.051883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:59.052071 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:59.052565 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:59.052573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:59.052952 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 16:34:59.053184 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 16:34:59.053722 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":342,"threads_started":1}
I20260929 16:34:59.056319 18736 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33037
I20260929 16:34:59.056679 18736 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:59.057482 18736 heartbeater.cc:511] Master 127.17.205.62:33037 requested a full tablet report, sending...
I20260929 16:34:59.058977 18280 ts_manager.cc:194] Registered new tserver with Master: c0308af6584b4e1aa3409b5f4ce29051 (127.17.205.3:34871)
I20260929 16:34:59.059832 18280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:54177
I20260929 16:34:59.064311 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:34:59.108114 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.093120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.17.205.1:36879)
0929 16:34:59.093766 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.093775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.093878 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:34:59.094056 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.094062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.094274 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.094409 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.095845 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.096832 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.098138 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.098797 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.098911 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.102734 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.102747 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.102764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.102801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.105311 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.105972 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.105980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.105982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.106107 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.106493 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.106497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.106499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.106848 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.107127 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.107514 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.107777 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
W20260929 16:34:59.109396 18384 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53226
I20260929 16:34:59.119380 18469 heartbeater.cc:503] Master 127.17.205.62:33037 was elected leader, sending a full tablet report...
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:34:59.151813 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.138645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.17.205.1:36879)
0929 16:34:59.138877 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.138883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.139087 (+ 204us) server_negotiation.cc:407] Connection header received
0929 16:34:59.139327 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.139332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.139419 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.139503 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.140893 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.141606 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.142782 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.143298 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.143398 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.147015 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.147021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.147024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.147060 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.149511 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.150035 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.150038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.150040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.150187 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.150516 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.150521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.150523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.150818 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.151004 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.151291 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.151472 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:34:59.153273 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18341
I20260929 16:34:59.164933 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.154262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.17.205.1:36879)
0929 16:34:59.154437 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.154441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.154639 (+ 198us) server_negotiation.cc:407] Connection header received
0929 16:34:59.154753 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.154756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.154810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.154885 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.155995 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.156771 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.157951 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.158349 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.158442 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.160412 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.160416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.160420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.160448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.162838 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.163404 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.163411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.163414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.163551 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.163867 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.163871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.163873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.164166 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.164352 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.164525 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.164683 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:34:59.165607 18464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:59.246520 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18341
I20260929 16:34:59.259661 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18475
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:34:59.274996 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.260574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.17.205.2:35907)
0929 16:34:59.261111 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.261118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.261145 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:34:59.261208 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.261214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.261417 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.261547 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.262670 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.263699 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.264883 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.265448 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.265554 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.269769 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.269781 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.269796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.269834 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.272401 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.273009 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.273016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.273019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.273135 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.273510 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.273514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.273516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.273849 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.274118 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.274414 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.274670 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
I20260929 16:34:59.276285 18598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:59.356492 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18475
I20260929 16:34:59.374440 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18608
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:34:59.389566 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.375436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.17.205.3:34871)
0929 16:34:59.376052 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:59.376059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:59.376086 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:34:59.376140 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:59.376147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:59.376373 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:59.376518 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:59.377686 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.378761 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.380214 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:59.380753 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:59.380848 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:59.384561 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:59.384573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:59.384588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:59.384621 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:59.387195 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.387829 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.387836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.387839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.387948 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:59.388237 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:59.388240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:59.388242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:59.388550 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:59.388773 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:59.389094 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:34:59.389296 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":325,"threads_started":1}
I20260929 16:34:59.390846 18731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:59.470295 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18608
I20260929 16:34:59.483325 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18248
I20260929 16:34:59.484488 18310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:59.583837 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18248
2026-09-29T16:34:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3634 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18757](info): set up 2 sockets
Sep 29 16:34:59 dist-test-slave-3h01 krb5kdc[18757](info): commencing operation
krb5kdc: starting...
W20260929 16:35:00.562863 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:00 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:00.730438 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42703
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:42703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:01.039393 18770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:01.039703 18770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:01.039834 18770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:01.046991 18770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:01.047106 18770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:01.047150 18770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:01.047232 18770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:01.047294 18770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:01.057211 18770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:42703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:01.058557 18770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:01.059937 18770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:01.074975 18775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:01.075084 18776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:01.075117 18778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:01.076134 18770 server_base.cc:1034] running on GCE node
I20260929 16:35:01.076946 18770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:01.078456 18770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:01.079649 18770 hybrid_clock.cc:648] HybridClock initialized: now 1790699701079598 us; error 42 us; skew 500 ppm
I20260929 16:35:01.080006 18770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:01 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699701, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:01.082960 18770 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:01.084493 18770 webserver.cc:460] Webserver started at http://127.17.205.62:36913/ using document root <none> and password file <none>
I20260929 16:35:01.085177 18770 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:01.085295 18770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:01.085593 18770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:01.088580 18770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "b77570941c3445eaa46357226d636dd8"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "0ee92846f26a80a43bd383b3f5628ace"
server_key_iv: "3335116d5d115fa7adb8e4ec62325904"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.089296 18770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "b77570941c3445eaa46357226d636dd8"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "0ee92846f26a80a43bd383b3f5628ace"
server_key_iv: "3335116d5d115fa7adb8e4ec62325904"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.094367 18770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:35:01.097767 18785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:01.099118 18770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:01.099310 18770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "b77570941c3445eaa46357226d636dd8"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "0ee92846f26a80a43bd383b3f5628ace"
server_key_iv: "3335116d5d115fa7adb8e4ec62325904"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.099491 18770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:01.116370 18770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:01.120894 18770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:01.121178 18770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:01.139883 18770 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:42703
I20260929 16:35:01.139889 18836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:42703 every 8 connection(s)
I20260929 16:35:01.141292 18770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:01.145085 18837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:01.146131 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18770
I20260929 16:35:01.146328 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:01.146951 18228 external_mini_cluster.cc:1428] Setting key 24c3026cd840aa8e11f9a999df48a0e4
I20260929 16:35:01.157575 18837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8: Bootstrap starting.
Sep 29 16:35:01 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:01.160935 18837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:01.162079 18837 log.cc:824] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:01.164983 18837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8: No bootstrap required, opened a new log
I20260929 16:35:01.168332 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:01.149051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.17.205.62:42703)
0929 16:35:01.149895 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:01.149907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:01.149951 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:35:01.151548 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:01.151577 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:01.152006 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:01.152384 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:01.153773 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.155079 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.156307 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.156968 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.157127 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:01.161066 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:01.161075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:01.161091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:01.161131 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:01.164277 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:01.164912 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:01.164919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:01.164926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:01.165154 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:01.165499 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:01.165504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:01.165507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:01.166045 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:01.166314 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:01.166853 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 16:35:01.167296 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":368,"threads_started":1}
I20260929 16:35:01.172811 18837 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } }
I20260929 16:35:01.173188 18837 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:01.173280 18837 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77570941c3445eaa46357226d636dd8, State: Initialized, Role: FOLLOWER
I20260929 16:35:01.173924 18837 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } }
I20260929 16:35:01.174177 18837 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:01.174314 18837 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:01.174480 18837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:01.176641 18837 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } }
I20260929 16:35:01.177091 18837 leader_election.cc:302] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77570941c3445eaa46357226d636dd8; no voters:
I20260929 16:35:01.177630 18837 leader_election.cc:288] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:01.177776 18842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:01.178411 18842 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [term 1 LEADER]: Becoming Leader. State: Replica: b77570941c3445eaa46357226d636dd8, State: Running, Role: LEADER
I20260929 16:35:01.178929 18842 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } }
I20260929 16:35:01.179855 18837 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:01.182008 18843 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77570941c3445eaa46357226d636dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } } }
I20260929 16:35:01.181996 18844 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77570941c3445eaa46357226d636dd8. Latest consensus state: current_term: 1 leader_uuid: "b77570941c3445eaa46357226d636dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77570941c3445eaa46357226d636dd8" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42703 } } }
I20260929 16:35:01.182464 18844 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:01.182472 18843 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:01.184007 18848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:01.189333 18848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:01.198987 18848 catalog_manager.cc:1347] Generated new cluster ID: 543ff3d722a0427d9f6293aade5e8632
I20260929 16:35:01.199115 18848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:01.213047 18848 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:01.214510 18848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:01.222076 18848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77570941c3445eaa46357226d636dd8: Generated new TSK 0
I20260929 16:35:01.222947 18848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:01.275154 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:01.606820 18863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:01.607122 18863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:01.607311 18863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:01.614841 18863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:01.614977 18863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:01.615131 18863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:01.625205 18863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:01.626477 18863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:01.627882 18863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:01.640599 18868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:01.640688 18869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:01.641104 18863 server_base.cc:1034] running on GCE node
W20260929 16:35:01.640650 18871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:01.641868 18863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:01.642810 18863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:01.643996 18863 hybrid_clock.cc:648] HybridClock initialized: now 1790699701643943 us; error 40 us; skew 500 ppm
I20260929 16:35:01.644322 18863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:01 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699701, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:01.647202 18863 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:01.648634 18863 webserver.cc:460] Webserver started at http://127.17.205.1:43175/ using document root <none> and password file <none>
I20260929 16:35:01.649351 18863 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:01.649466 18863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:01.649816 18863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:01.652860 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "f0783ccdccc84479bd6201be8d8fbc7b"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "278c7bb2365127c5a0e7cd7f4ac2c662"
server_key_iv: "6543ba9581a15788589f130e3053b5b0"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.653610 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "f0783ccdccc84479bd6201be8d8fbc7b"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "278c7bb2365127c5a0e7cd7f4ac2c662"
server_key_iv: "6543ba9581a15788589f130e3053b5b0"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.658605 18863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:01.662106 18878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:01.663525 18863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:01.663697 18863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "f0783ccdccc84479bd6201be8d8fbc7b"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-3h01"
server_key: "278c7bb2365127c5a0e7cd7f4ac2c662"
server_key_iv: "6543ba9581a15788589f130e3053b5b0"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.663870 18863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:01.679909 18863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:01.683908 18863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:01.684167 18863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:01.685127 18863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:01.686420 18863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:01.686493 18863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:01.686607 18863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:01.686668 18863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:01.714818 18863 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42853
I20260929 16:35:01.714860 18990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42853 every 8 connection(s)
I20260929 16:35:01.716123 18863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:01.722640 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18863
I20260929 16:35:01.722858 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:01.723609 18228 external_mini_cluster.cc:1428] Setting key 0da651981c7b0def8acde75560e8ec48
Sep 29 16:35:01 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699701, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:01.736212 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:01.718607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:50169 (local address 127.17.205.62:42703)
0929 16:35:01.719012 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:01.719016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:01.719994 (+ 978us) server_negotiation.cc:407] Connection header received
0929 16:35:01.721805 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:01.721811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:01.721892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:01.721986 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:01.724060 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.724968 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.726656 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.727267 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.727385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:01.730750 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:01.730755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:01.730758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:01.730790 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:01.733158 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:01.733968 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:01.733972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:01.733974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:01.734082 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:01.734531 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:01.734534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:01.734536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:01.734805 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:01.734921 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:01.735793 (+ 872us) server_negotiation.cc:299] Negotiation successful
0929 16:35:01.735956 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260929 16:35:01.737742 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:01.719101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42703 (local address 127.17.205.1:50169)
0929 16:35:01.719835 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:01.719882 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:01.721540 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:01.722259 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:01.722271 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:01.722997 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:01.723875 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:01.723902 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.725129 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.725134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:01.726496 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:01.726508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.727457 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.727463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:01.727654 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:01.728266 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:01.728288 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:01.730535 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:01.733290 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:01.733300 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:01.733319 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:01.733869 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:01.734174 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:01.734178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:01.734182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:01.734430 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:01.734925 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:01.734934 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:01.735406 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 16:35:01.735733 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 16:35:01.736285 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:35:01.739333 18991 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42703
I20260929 16:35:01.739640 18991 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:01.740358 18991 heartbeater.cc:511] Master 127.17.205.62:42703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:35:01.742400 18802 ts_manager.cc:194] Registered new tserver with Master: f0783ccdccc84479bd6201be8d8fbc7b (127.17.205.1:42853)
I20260929 16:35:01.743860 18802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:50169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:01.763042 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:02.075698 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:02.076030 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:02.076184 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:02.083777 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:02.083933 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:02.084111 18997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:02.094163 18997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:02.095539 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:02.096971 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:02.110409 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:02.110440 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:02.110387 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:02.111028 18997 server_base.cc:1034] running on GCE node
I20260929 16:35:02.111747 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:02.112622 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:02.113813 18997 hybrid_clock.cc:648] HybridClock initialized: now 1790699702113760 us; error 44 us; skew 500 ppm
I20260929 16:35:02.114120 18997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:02.117156 18997 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:02.118574 18997 webserver.cc:460] Webserver started at http://127.17.205.2:45317/ using document root <none> and password file <none>
I20260929 16:35:02.119277 18997 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:02.119378 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:02.119660 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:02.122495 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "01a58c9b7a884f248e394f383d593d4b"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "18c38e8d0e6c4fa04c98b9e7098da8d0"
server_key_iv: "27d1cca930650ab814237a2a66ea2887"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.123306 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "01a58c9b7a884f248e394f383d593d4b"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "18c38e8d0e6c4fa04c98b9e7098da8d0"
server_key_iv: "27d1cca930650ab814237a2a66ea2887"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.128743 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:02.132764 19012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.134150 18997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:35:02.134294 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "01a58c9b7a884f248e394f383d593d4b"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "18c38e8d0e6c4fa04c98b9e7098da8d0"
server_key_iv: "27d1cca930650ab814237a2a66ea2887"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.134508 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:02.163909 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:02.168241 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:02.168509 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:02.169480 18997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:02.170797 18997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:02.170874 18997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.170989 18997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:02.171044 18997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.199448 18997 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:37069
I20260929 16:35:02.199488 19124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:37069 every 8 connection(s)
I20260929 16:35:02.200681 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:02.208106 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 18997
I20260929 16:35:02.208273 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:02.208678 18228 external_mini_cluster.cc:1428] Setting key 32e9a4a72446658a66b293cd23a782fa
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:02.220768 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.203115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:41587 (local address 127.17.205.62:42703)
0929 16:35:02.203308 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.203312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.204548 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 16:35:02.206392 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.206398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.206486 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.206571 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:02.208132 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.208927 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.210601 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.210996 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.211111 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.214596 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:02.214602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:02.214607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:02.214647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:02.217768 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.218440 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.218445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.218449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.218596 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.218978 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.218982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.218985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.219345 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:02.219481 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.220202 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.220425 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:35:02.221637 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.203701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42703 (local address 127.17.205.2:41587)
0929 16:35:02.204401 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.204445 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.206120 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.206821 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.206830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.207374 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:02.207964 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.207983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.209072 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.209076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.210477 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.210489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.211194 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.211201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.211439 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.212081 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:02.212105 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:02.214329 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:02.217891 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:02.217898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:02.217911 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:02.218329 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:02.218689 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:02.218692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:02.218695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:02.218877 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:02.219449 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:02.219455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:02.219834 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.220061 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.220476 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":253,"threads_started":1}
I20260929 16:35:02.223038 19125 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42703
I20260929 16:35:02.223362 19125 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:02.224056 19125 heartbeater.cc:511] Master 127.17.205.62:42703 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:35:02.225531 18802 ts_manager.cc:194] Registered new tserver with Master: 01a58c9b7a884f248e394f383d593d4b (127.17.205.2:37069)
I20260929 16:35:02.226327 18802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:41587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:02.246381 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:02.560971 19130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:02.561288 19130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:02.561431 19130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:02.568671 19130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:02.568810 19130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:02.568964 19130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:02.578840 19130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:02.580325 19130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:02.581754 19130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:02.595026 19135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:02.594995 19138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:02.594977 19136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:02.595865 19130 server_base.cc:1034] running on GCE node
I20260929 16:35:02.596503 19130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:02.597464 19130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:02.598690 19130 hybrid_clock.cc:648] HybridClock initialized: now 1790699702598621 us; error 64 us; skew 500 ppm
I20260929 16:35:02.599016 19130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:02.609021 19130 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:02.610710 19130 webserver.cc:460] Webserver started at http://127.17.205.3:34959/ using document root <none> and password file <none>
I20260929 16:35:02.611522 19130 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:02.611647 19130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:02.611950 19130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:02.614934 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "94fcef282afd4a7b81789037ff2bfe19"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "1a05834a0425ac92b8399e887c5ebeb7"
server_key_iv: "e8f4357d72ccf3bfa33236cce57ec5e6"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.615836 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "94fcef282afd4a7b81789037ff2bfe19"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "1a05834a0425ac92b8399e887c5ebeb7"
server_key_iv: "e8f4357d72ccf3bfa33236cce57ec5e6"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.621184 19130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:02.624940 19145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.626482 19130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:35:02.626650 19130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "94fcef282afd4a7b81789037ff2bfe19"
format_stamp: "Formatted at 2026-09-29 16:35:02 on dist-test-slave-3h01"
server_key: "1a05834a0425ac92b8399e887c5ebeb7"
server_key_iv: "e8f4357d72ccf3bfa33236cce57ec5e6"
server_key_version: "encryptionkey@0"
I20260929 16:35:02.626839 19130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:02.653016 19130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:02.657104 19130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:02.657384 19130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:02.658437 19130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:02.659976 19130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:02.660050 19130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.660175 19130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:02.660233 19130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:02.689550 19130 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43483
I20260929 16:35:02.689592 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43483 every 8 connection(s)
I20260929 16:35:02.690856 19130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:02.692276 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19130
I20260929 16:35:02.692441 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:02.692836 18228 external_mini_cluster.cc:1428] Setting key 302fa9602e0f86b89213b4a25674949d
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:02.711756 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.693423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:39147 (local address 127.17.205.62:42703)
0929 16:35:02.693607 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.693613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.695127 (+ 1514us) server_negotiation.cc:407] Connection header received
0929 16:35:02.697246 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.697251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.697329 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.697413 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:02.699038 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.699827 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.701430 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.701823 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.701937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.705669 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:02.705674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:02.705678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:02.705717 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:02.708782 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.709546 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.709550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.709553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.709684 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.710113 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.710117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.710119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.710430 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:02.710551 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.711304 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.711482 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:35:02.712729 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.694161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42703 (local address 127.17.205.3:39147)
0929 16:35:02.694989 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.695029 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.696933 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.697690 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.697699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.698240 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:02.698865 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.698883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.699998 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.700002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.701279 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.701292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.702076 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.702083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.702325 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.703014 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:02.703043 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:02.705437 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:02.708922 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:02.708931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:02.708948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:02.709420 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:02.709783 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:02.709786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:02.709788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:02.709977 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:02.710544 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:02.710551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:02.710891 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.711131 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.711608 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":298,"threads_started":1}
I20260929 16:35:02.714169 19258 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42703
I20260929 16:35:02.714525 19258 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:02.715297 19258 heartbeater.cc:511] Master 127.17.205.62:42703 requested a full tablet report, sending...
I20260929 16:35:02.716794 18802 ts_manager.cc:194] Registered new tserver with Master: 94fcef282afd4a7b81789037ff2bfe19 (127.17.205.3:43483)
I20260929 16:35:02.717610 18802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:39147
I20260929 16:35:02.720821 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:02.738731 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.726774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.17.205.62:42703)
0929 16:35:02.726947 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.726952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.727388 (+ 436us) server_negotiation.cc:407] Connection header received
0929 16:35:02.727525 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.727529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.727594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.727689 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:02.729076 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.729881 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.731203 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.731689 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.731810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.733911 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:02.733916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:02.733920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:02.733951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:02.736401 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.736994 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.736997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.737000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.737121 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.737502 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.737506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.737508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.737854 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:02.738065 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.738325 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.738489 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:35:02.746834 18991 heartbeater.cc:503] Master 127.17.205.62:42703 was elected leader, sending a full tablet report...
I20260929 16:35:02.744958 18802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:02.752063 18802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:02.774338 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.764759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.17.205.3:43483)
0929 16:35:02.766280 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.766288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.766319 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:02.767183 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.767192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.767458 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.767715 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:02.768349 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.769938 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.772095 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.773339 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.773448 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.773676 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.773909 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.774052 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":1665,"threads_started":1}
I20260929 16:35:02.774320 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.764129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.17.205.1:42853)
0929 16:35:02.764857 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.764867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.765506 (+ 639us) server_negotiation.cc:407] Connection header received
0929 16:35:02.765856 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.765866 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.766133 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.766310 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:02.767056 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.768596 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.771462 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.772885 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.773012 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.773255 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.773820 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.773988 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":333,"threads_started":1}
I20260929 16:35:02.774852 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.766316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:37069 (local address 127.0.0.1:54808)
0929 16:35:02.767478 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.767494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.767739 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.768262 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.768266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.768288 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:02.768601 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.768612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.770205 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.770210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.772038 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.772051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.774098 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.774104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.774260 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.774293 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.774380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.774499 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":4,"spinlock_wait_cycles":8064,"thread_start_us":941,"threads_started":1}
I20260929 16:35:02.775017 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.766544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.17.205.2:37069)
0929 16:35:02.767015 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.767022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.767634 (+ 612us) server_negotiation.cc:407] Connection header received
0929 16:35:02.767933 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.767941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.768109 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.768261 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:02.768792 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.770065 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.772356 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.773660 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.773792 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.773960 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.774648 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.774781 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":254,"threads_started":1}
I20260929 16:35:02.774349 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.763957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42853 (local address 127.0.0.1:43268)
0929 16:35:02.765365 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.765400 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.765647 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.766427 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.766435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.766465 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:02.766843 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.766857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.768977 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.768982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.771299 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.771314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.773369 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.773376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.773574 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.773629 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.773762 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.773957 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":1041,"threads_started":1}
I20260929 16:35:02.776240 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.764253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43483 (local address 127.0.0.1:57602)
0929 16:35:02.766169 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.766184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.767024 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.767678 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.767683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.767724 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:02.768110 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.768122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.770091 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.770096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.771933 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.771949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.773524 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.773530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.773697 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.773728 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.773809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.774886 (+ 1077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":1013,"spinlock_wait_cycles":4608,"thread_start_us":1647,"threads_started":1}
I20260929 16:35:02.778642 18926 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c4070d449c894447f29631ae8dc (DEFAULT_TABLE table=test-table [id=e5eabdb3faa24f6aa3d9d045bca79731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:02.778721 19060 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c4070d449c894447f29631ae8dc (DEFAULT_TABLE table=test-table [id=e5eabdb3faa24f6aa3d9d045bca79731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:02.779976 18926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c4070d449c894447f29631ae8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:02.779548 19193 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c4070d449c894447f29631ae8dc (DEFAULT_TABLE table=test-table [id=e5eabdb3faa24f6aa3d9d045bca79731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:02.780087 19060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c4070d449c894447f29631ae8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:02.780794 19193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c4070d449c894447f29631ae8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:02.791458 19276 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Bootstrap starting.
I20260929 16:35:02.794394 19277 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Bootstrap starting.
I20260929 16:35:02.795998 19278 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Bootstrap starting.
I20260929 16:35:02.796350 19276 tablet_bootstrap.cc:654] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:02.797684 19276 log.cc:824] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:02.797891 19277 tablet_bootstrap.cc:654] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:02.798873 19278 tablet_bootstrap.cc:654] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:02.798941 19277 log.cc:824] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:02.800047 19278 log.cc:824] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:02.800963 19276 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: No bootstrap required, opened a new log
I20260929 16:35:02.801209 19276 ts_tablet_manager.cc:1397] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 16:35:02.801717 19277 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: No bootstrap required, opened a new log
I20260929 16:35:02.801975 19277 ts_tablet_manager.cc:1397] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:35:02.802910 19278 tablet_bootstrap.cc:492] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: No bootstrap required, opened a new log
I20260929 16:35:02.803156 19278 ts_tablet_manager.cc:1397] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 16:35:02.808921 19276 raft_consensus.cc:348] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.809334 19276 raft_consensus.cc:374] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:02.809429 19276 raft_consensus.cc:729] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0783ccdccc84479bd6201be8d8fbc7b, State: Initialized, Role: FOLLOWER
I20260929 16:35:02.810125 19276 consensus_queue.cc:260] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.811811 19276 ts_tablet_manager.cc:1428] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260929 16:35:02.813203 19277 raft_consensus.cc:348] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.813213 19278 raft_consensus.cc:348] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.813562 19277 raft_consensus.cc:374] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:02.813648 19277 raft_consensus.cc:729] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcef282afd4a7b81789037ff2bfe19, State: Initialized, Role: FOLLOWER
I20260929 16:35:02.813647 19278 raft_consensus.cc:374] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:02.813746 19278 raft_consensus.cc:729] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a58c9b7a884f248e394f383d593d4b, State: Initialized, Role: FOLLOWER
I20260929 16:35:02.814265 19277 consensus_queue.cc:260] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.814318 19278 consensus_queue.cc:260] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.816838 19258 heartbeater.cc:503] Master 127.17.205.62:42703 was elected leader, sending a full tablet report...
I20260929 16:35:02.817194 19125 heartbeater.cc:503] Master 127.17.205.62:42703 was elected leader, sending a full tablet report...
I20260929 16:35:02.817572 19277 ts_tablet_manager.cc:1428] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260929 16:35:02.817866 19278 ts_tablet_manager.cc:1428] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260929 16:35:02.872407 19283 raft_consensus.cc:482] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:02.872678 19283 raft_consensus.cc:504] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.874307 19283 leader_election.cc:288] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0783ccdccc84479bd6201be8d8fbc7b (127.17.205.1:42853), 94fcef282afd4a7b81789037ff2bfe19 (127.17.205.3:43483)
I20260929 16:35:02.880746 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.874835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:35667 (local address 127.17.205.3:43483)
0929 16:35:02.875079 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.875085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.875605 (+ 520us) server_negotiation.cc:407] Connection header received
0929 16:35:02.875779 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.875785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.875846 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.875910 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:02.876525 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.877255 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.879042 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.879852 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.880000 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.880076 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.880380 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.880501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 16:35:02.880915 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.874650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42853 (local address 127.17.205.2:54581)
0929 16:35:02.875539 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.875554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.875683 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.876083 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.876087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.876126 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:02.876506 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.876515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.877483 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.877487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.878903 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.878915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.880163 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.880171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.880300 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.880321 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.880400 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.880523 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":390,"threads_started":1}
I20260929 16:35:02.880972 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.874637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43483 (local address 127.17.205.2:35667)
0929 16:35:02.875475 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:02.875495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:02.875642 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:02.875947 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:02.875952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:02.875983 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:02.876374 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.876387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.877378 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.877383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.878888 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:02.878900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.880058 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.880064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:02.880222 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.880257 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:35:02.880338 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:35:02.880473 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":1,"spinlock_wait_cycles":22400,"thread_start_us":632,"threads_started":1}
I20260929 16:35:02.880788 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.874876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:54581 (local address 127.17.205.1:42853)
0929 16:35:02.875056 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.875062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.875654 (+ 592us) server_negotiation.cc:407] Connection header received
0929 16:35:02.875807 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.875811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.875869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.875967 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:02.876652 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.877350 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.879058 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.879856 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.880005 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.880078 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.880445 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.880566 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:35:02.881959 19213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c4070d449c894447f29631ae8dc" candidate_uuid: "01a58c9b7a884f248e394f383d593d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcef282afd4a7b81789037ff2bfe19" is_pre_election: true
I20260929 16:35:02.882057 18946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c4070d449c894447f29631ae8dc" candidate_uuid: "01a58c9b7a884f248e394f383d593d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" is_pre_election: true
I20260929 16:35:02.882447 19213 raft_consensus.cc:2454] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a58c9b7a884f248e394f383d593d4b in term 0.
I20260929 16:35:02.882521 18946 raft_consensus.cc:2454] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a58c9b7a884f248e394f383d593d4b in term 0.
I20260929 16:35:02.883129 19014 leader_election.cc:302] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01a58c9b7a884f248e394f383d593d4b, 94fcef282afd4a7b81789037ff2bfe19, f0783ccdccc84479bd6201be8d8fbc7b; no voters:
I20260929 16:35:02.883579 19283 raft_consensus.cc:2781] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:02.883708 19283 raft_consensus.cc:482] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:02.883809 19283 raft_consensus.cc:3037] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:02.886068 19283 raft_consensus.cc:504] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.886628 19283 leader_election.cc:288] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [CANDIDATE]: Term 1 election: Requested vote from peers f0783ccdccc84479bd6201be8d8fbc7b (127.17.205.1:42853), 94fcef282afd4a7b81789037ff2bfe19 (127.17.205.3:43483)
I20260929 16:35:02.887173 19213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c4070d449c894447f29631ae8dc" candidate_uuid: "01a58c9b7a884f248e394f383d593d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcef282afd4a7b81789037ff2bfe19"
I20260929 16:35:02.887161 18946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c4070d449c894447f29631ae8dc" candidate_uuid: "01a58c9b7a884f248e394f383d593d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0783ccdccc84479bd6201be8d8fbc7b"
I20260929 16:35:02.887398 18946 raft_consensus.cc:3037] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:02.887398 19213 raft_consensus.cc:3037] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:02.890465 19213 raft_consensus.cc:2454] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a58c9b7a884f248e394f383d593d4b in term 1.
I20260929 16:35:02.890465 18946 raft_consensus.cc:2454] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a58c9b7a884f248e394f383d593d4b in term 1.
I20260929 16:35:02.890883 19014 leader_election.cc:302] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a58c9b7a884f248e394f383d593d4b, f0783ccdccc84479bd6201be8d8fbc7b; no voters:
I20260929 16:35:02.891179 19283 raft_consensus.cc:2781] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:02.891683 19283 raft_consensus.cc:686] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [term 1 LEADER]: Becoming Leader. State: Replica: 01a58c9b7a884f248e394f383d593d4b, State: Running, Role: LEADER
I20260929 16:35:02.892155 19283 consensus_queue.cc:237] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } }
I20260929 16:35:02.896876 18802 catalog_manager.cc:5495] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a58c9b7a884f248e394f383d593d4b (127.17.205.2). New cstate: current_term: 1 leader_uuid: "01a58c9b7a884f248e394f383d593d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0783ccdccc84479bd6201be8d8fbc7b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94fcef282afd4a7b81789037ff2bfe19" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a58c9b7a884f248e394f383d593d4b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 37069 } health_report { overall_health: HEALTHY } } }
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:02.935541 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.921088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.17.205.1:42853)
0929 16:35:02.921302 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.921310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.921646 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:35:02.921915 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.921920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.921992 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.922078 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:02.923692 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.924503 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.925715 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.926134 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.926237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.930127 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:02.930145 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:02.930162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:02.930208 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:02.932988 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.933694 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.933703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.933706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.933863 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.934182 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.934186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.934188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.934564 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:02.934785 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.934983 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.935142 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:35:02.942624 19259 tablet.cc:2369] T be1c8c4070d449c894447f29631ae8dc P 94fcef282afd4a7b81789037ff2bfe19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:02.952579 19126 tablet.cc:2369] T be1c8c4070d449c894447f29631ae8dc P 01a58c9b7a884f248e394f383d593d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 16:35:02.968433 18992 tablet.cc:2369] T be1c8c4070d449c894447f29631ae8dc P f0783ccdccc84479bd6201be8d8fbc7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:35:02 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:02.981951 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:02.968039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.17.205.1:42853)
0929 16:35:02.968218 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:02.968225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:02.968647 (+ 422us) server_negotiation.cc:407] Connection header received
0929 16:35:02.968878 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:02.968884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:02.968968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:02.969068 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:02.970400 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.971133 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.972458 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:02.972900 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:02.973008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:02.976741 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:02.976747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:02.976751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:02.976797 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:02.979570 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.980241 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.980246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.980248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.980420 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:02.980806 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:02.980810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:02.980812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:02.981087 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:02.981284 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:02.981496 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:35:02.981668 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[18757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:03.023762 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.010057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.17.205.1:42853)
0929 16:35:03.010282 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.010288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.010569 (+ 281us) server_negotiation.cc:407] Connection header received
0929 16:35:03.010743 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.010747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.010813 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.010894 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.012242 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.012965 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.014172 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.014600 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.014716 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:03.018543 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.018549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.018552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:03.018606 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.021082 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.021784 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.021788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.021791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.021947 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.022320 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.022325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.022329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.022619 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.022861 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.023150 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.023408 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:35:03.025652 18906 tablet_service.cc:2908] Rejecting scan request for tablet be1c8c4070d449c894447f29631ae8dc: Uninitialized: clean time has not yet been initialized
I20260929 16:35:03.028486 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18863
I20260929 16:35:03.042215 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.029417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.17.205.1:42853)
0929 16:35:03.029584 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.029588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.029937 (+ 349us) server_negotiation.cc:407] Connection header received
0929 16:35:03.030064 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.030067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.030175 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.030283 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.031518 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.032242 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.033492 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.034029 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.034142 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:03.036588 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.036594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.036597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:03.036648 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.039692 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.040395 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.040399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.040401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.040532 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.040855 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.040858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.040860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.041183 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.041407 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.041673 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.041865 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:35:03.042825 18986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:03.130300 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18863
I20260929 16:35:03.145920 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18997
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:03.160445 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.146770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.17.205.2:37069)
0929 16:35:03.146981 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.146987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.147008 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:35:03.147068 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.147073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.147179 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.147298 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.148379 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.149473 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.150713 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.151409 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.151536 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:03.155353 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.155373 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.155391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:03.155448 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.158012 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.158654 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.158662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.158664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.158802 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.159088 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.159091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.159093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.159502 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.159719 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.159939 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.160116 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:35:03.161140 19120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:03.250349 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18997
I20260929 16:35:03.263809 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19130
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[18757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:03.278666 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.264686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.17.205.3:43483)
0929 16:35:03.264869 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.264875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.264896 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:35:03.265036 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.265041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.265113 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.265222 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.266400 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.267106 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.268428 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:03.268830 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:03.268944 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:03.272947 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.272963 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.272980 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:03.273016 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.275985 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.276684 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.276692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.276695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.276825 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:03.277190 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:03.277194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:03.277196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:03.277550 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.277807 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.278139 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.278355 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:35:03.279390 19253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:03.364670 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19130
I20260929 16:35:03.377452 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 18770
I20260929 16:35:03.378628 18832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:03.484546 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 18770
2026-09-29T16:35:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3902 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[19306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[19306](info): set up 2 sockets
Sep 29 16:35:03 dist-test-slave-3h01 krb5kdc[19306](info): commencing operation
krb5kdc: starting...
W20260929 16:35:04.471104 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:04 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:04.642283 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40121
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:04.950657 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:04.950994 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:04.951128 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:04.958570 19319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:04.958706 19319 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:35:04.958774 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:04.958858 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:04.958921 19319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:04.959002 19319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:04.969090 19319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:40121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:04.970418 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:04.971822 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:04.983512 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:04.983479 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:04.983451 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:04.984339 19319 server_base.cc:1034] running on GCE node
I20260929 16:35:04.985152 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:04.986624 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:04.987875 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790699704987803 us; error 37 us; skew 500 ppm
I20260929 16:35:04.988251 19319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:04 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:04.991731 19319 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:04.993252 19319 webserver.cc:460] Webserver started at http://127.17.205.62:42943/ using document root <none> and password file <none>
I20260929 16:35:04.993988 19319 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:04.994096 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:04.994424 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:04.997644 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "278d26fe848848e18912ba47527a01ae"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-3h01"
server_key: "14812498df811c908c1fee5fa709b54e"
server_key_iv: "5a75ef716ad2ea102699be112962cabd"
server_key_version: "encryptionkey@0"
I20260929 16:35:04.998466 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "278d26fe848848e18912ba47527a01ae"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-3h01"
server_key: "14812498df811c908c1fee5fa709b54e"
server_key_iv: "5a75ef716ad2ea102699be112962cabd"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.004294 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:35:05.008080 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:05.009572 19319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:35:05.009763 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "278d26fe848848e18912ba47527a01ae"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-3h01"
server_key: "14812498df811c908c1fee5fa709b54e"
server_key_iv: "5a75ef716ad2ea102699be112962cabd"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.009984 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:05.026051 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:05.030484 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:05.030761 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:05.050958 19319 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:40121
I20260929 16:35:05.050976 19395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:40121 every 8 connection(s)
I20260929 16:35:05.052414 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:05.056512 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:05.058476 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19319
I20260929 16:35:05.058683 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:05.059352 18228 external_mini_cluster.cc:1428] Setting key 3eab0eb2f5ab36baa635c4758d239f64
I20260929 16:35:05.068363 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Bootstrap starting.
I20260929 16:35:05.072207 19396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:05.073498 19396 log.cc:824] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:35:05 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:05.076903 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: No bootstrap required, opened a new log
I20260929 16:35:05.083081 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:05.061500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.17.205.62:40121)
0929 16:35:05.062285 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:05.062299 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:05.062348 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:35:05.064088 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:05.064114 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:05.064539 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:05.064893 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:05.066526 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.067888 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.069547 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.070254 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.070394 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:05.075002 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:05.075014 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:05.075035 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:05.075078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:05.078666 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.079334 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.079347 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.079356 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.079565 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.079900 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.079906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.079909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.080630 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:05.080919 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:05.081210 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:35:05.081659 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":365,"threads_started":1}
I20260929 16:35:05.086745 19396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } }
I20260929 16:35:05.087091 19396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:05.087167 19396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278d26fe848848e18912ba47527a01ae, State: Initialized, Role: FOLLOWER
I20260929 16:35:05.087716 19396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } }
I20260929 16:35:05.087917 19396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:05.088012 19396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:05.088127 19396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:05.090178 19396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } }
I20260929 16:35:05.090646 19396 leader_election.cc:302] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278d26fe848848e18912ba47527a01ae; no voters:
I20260929 16:35:05.091245 19396 leader_election.cc:288] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:05.091355 19401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:05.092211 19401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [term 1 LEADER]: Becoming Leader. State: Replica: 278d26fe848848e18912ba47527a01ae, State: Running, Role: LEADER
I20260929 16:35:05.092751 19401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } }
I20260929 16:35:05.093429 19396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:05.095454 19403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278d26fe848848e18912ba47527a01ae. Latest consensus state: current_term: 1 leader_uuid: "278d26fe848848e18912ba47527a01ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } } }
I20260929 16:35:05.095860 19403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:05.097177 19402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278d26fe848848e18912ba47527a01ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278d26fe848848e18912ba47527a01ae" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40121 } } }
I20260929 16:35:05.097646 19402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:05.098819 19408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:05.103946 19408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:05.113057 19408 catalog_manager.cc:1347] Generated new cluster ID: 234ddad7c82a4d868d49bab12c99f31a
I20260929 16:35:05.113240 19408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:05.127933 19408 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:05.129266 19408 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:05.135790 19408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Generated new TSK 0
I20260929 16:35:05.136620 19408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:05.184439 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40121
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:05.505530 19422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:05.505837 19422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:05.505934 19422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:05.513266 19422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:05.513415 19422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:05.513542 19422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:05.513652 19422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:05.523645 19422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:05.525031 19422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:05.526398 19422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:05.539644 19427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:05.539754 19428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:05.539754 19430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:05.540572 19422 server_base.cc:1034] running on GCE node
I20260929 16:35:05.541165 19422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:05.542125 19422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:05.543318 19422 hybrid_clock.cc:648] HybridClock initialized: now 1790699705543249 us; error 43 us; skew 500 ppm
I20260929 16:35:05.543618 19422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:05 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699705, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:05.546693 19422 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:05.548290 19422 webserver.cc:460] Webserver started at http://127.17.205.1:46007/ using document root <none> and password file <none>
I20260929 16:35:05.548998 19422 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:05.549085 19422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:05.549399 19422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:05.552453 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "3b9422ad2bde40f6bb223aa8f9065407"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-3h01"
server_key: "4d3b3b54e8bed7f508d4f064ae742e71"
server_key_iv: "74bf1f8b4796553fb25a7eabcbcbeda6"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.553236 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "3b9422ad2bde40f6bb223aa8f9065407"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-3h01"
server_key: "4d3b3b54e8bed7f508d4f064ae742e71"
server_key_iv: "74bf1f8b4796553fb25a7eabcbcbeda6"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.558634 19422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:05.562455 19437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:05.563959 19422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:05.564122 19422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "3b9422ad2bde40f6bb223aa8f9065407"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-3h01"
server_key: "4d3b3b54e8bed7f508d4f064ae742e71"
server_key_iv: "74bf1f8b4796553fb25a7eabcbcbeda6"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.564324 19422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:05.583372 19422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:05.587899 19422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:05.588147 19422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:05.590869 19422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:05.590965 19422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:05.591104 19422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:05.591161 19422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:05 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699705, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:05.615926 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:05.591440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:50123 (local address 127.17.205.62:40121)
0929 16:35:05.592079 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:05.592092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:05.593294 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 16:35:05.595816 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:05.595822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:05.595933 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:05.596051 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:05.598791 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.599767 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.602950 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.603572 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.603714 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:05.608313 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:05.608322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:05.608327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:05.608386 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:05.611875 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.613000 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.613006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.613010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.613197 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.613832 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.613837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.613841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.614195 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:05.614320 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:05.615486 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0929 16:35:05.615671 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":367,"threads_started":1}
I20260929 16:35:05.618014 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:05.592225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.1:50123)
0929 16:35:05.593098 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:05.593147 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:05.595379 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:05.596416 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:05.596433 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:05.597451 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:05.598507 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:05.598540 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.599987 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.599995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:05.602744 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:05.602768 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.604056 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.604067 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:05.604340 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:05.605160 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:05.605187 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:05.608009 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:05.612042 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:05.612056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:05.612081 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:05.612806 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:05.613338 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:05.613344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:05.613348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:05.613697 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:05.614354 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:05.614368 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:05.614971 (+ 603us) client_negotiation.cc:769] Sending connection context
0929 16:35:05.615419 (+ 448us) client_negotiation.cc:240] Negotiation successful
0929 16:35:05.616076 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":341,"threads_started":1}
I20260929 16:35:05.635398 19422 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:39505
I20260929 16:35:05.635658 19557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:39505 every 8 connection(s)
I20260929 16:35:05.637041 19422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:05.639132 19558 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40121
I20260929 16:35:05.639184 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:05.625171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.1:60667)
0929 16:35:05.625859 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:05.625881 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:05.626043 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:05.626352 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:05.626357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:05.626981 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:05.627401 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:05.627414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.628374 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.628379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:05.630066 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:05.630082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.630834 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.630840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:05.631032 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:05.631801 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:05.631830 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:05.632902 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:05.636478 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:05.636483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:05.636487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:05.636883 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:05.637239 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:05.637244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:05.637248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:05.637402 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:05.638063 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:05.638068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:05.638292 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 16:35:05.638427 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:35:05.638768 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":366,"threads_started":1}
I20260929 16:35:05.639546 19558 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:05.639878 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:05.625060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:60667 (local address 127.17.205.62:40121)
0929 16:35:05.625274 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:05.625281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:05.625974 (+ 693us) server_negotiation.cc:407] Connection header received
0929 16:35:05.626151 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:05.626155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:05.626224 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:05.626302 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:05.627539 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.628250 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.630214 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:05.630644 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:05.630765 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:05.633106 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:05.633111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:05.633114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:05.633146 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:05.636362 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.636990 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.636996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.637000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.637128 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:05.637510 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:05.637515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:05.637518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:05.637844 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:05.637997 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:05.639364 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0929 16:35:05.639587 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:35:05.640529 19558 heartbeater.cc:511] Master 127.17.205.62:40121 requested a full tablet report, sending...
I20260929 16:35:05.641355 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19422
I20260929 16:35:05.641549 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:05.642299 18228 external_mini_cluster.cc:1428] Setting key 6711117ec294fddf22feda4e845e045b
I20260929 16:35:05.642863 19351 ts_manager.cc:194] Registered new tserver with Master: 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505)
I20260929 16:35:05.644560 19351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:50123
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:05.682169 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40121
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:05.995355 19562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:05.995645 19562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:05.995764 19562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:06.002995 19562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:06.003115 19562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:06.003244 19562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:06.003350 19562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:06.013207 19562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:06.014514 19562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:06.015918 19562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:06.028730 19567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:06.028884 19570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:06.028913 19568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:06.029856 19562 server_base.cc:1034] running on GCE node
I20260929 16:35:06.030481 19562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:06.031489 19562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:06.032704 19562 hybrid_clock.cc:648] HybridClock initialized: now 1790699706032643 us; error 54 us; skew 500 ppm
I20260929 16:35:06.032994 19562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:06.036069 19562 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:06.037647 19562 webserver.cc:460] Webserver started at http://127.17.205.2:37257/ using document root <none> and password file <none>
I20260929 16:35:06.038357 19562 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:06.038444 19562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:06.038746 19562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:06.041730 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "db58384b737a488487a76a480acd4a2c"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "e26039f8e4470ffd223032e014895697"
server_key_iv: "c6f2a1f6cf228e3bed71cdd9d42508a3"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.042524 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "db58384b737a488487a76a480acd4a2c"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "e26039f8e4470ffd223032e014895697"
server_key_iv: "c6f2a1f6cf228e3bed71cdd9d42508a3"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.047786 19562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:35:06.051517 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:06.053081 19562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:06.053251 19562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "db58384b737a488487a76a480acd4a2c"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "e26039f8e4470ffd223032e014895697"
server_key_iv: "c6f2a1f6cf228e3bed71cdd9d42508a3"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.053478 19562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:06.071012 19562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:06.074924 19562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:06.075150 19562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:06.077705 19562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:06.077804 19562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:06.077945 19562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:06.078009 19562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:06.101145 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.078563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:59863 (local address 127.17.205.62:40121)
0929 16:35:06.078738 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.078745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.080252 (+ 1507us) server_negotiation.cc:407] Connection header received
0929 16:35:06.083067 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.083073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.083164 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.083359 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.085458 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.086580 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.088764 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.089241 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.089374 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.094113 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.094120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.094125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:06.094163 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.097294 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.098287 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.098293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.098298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.098481 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.099022 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.099027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.099030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.099387 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.099528 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.100512 (+ 984us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.100766 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:35:06.102460 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.079182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.2:59863)
0929 16:35:06.080067 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.080124 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.082667 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.083607 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.083619 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.084383 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:06.085238 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.085273 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.086726 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.086731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.088626 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.088650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.089522 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.089529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.089847 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.090627 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:06.090662 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:06.093858 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:06.097470 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.097480 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.097503 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.098134 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.098598 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.098603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.098606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.098901 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.099520 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:06.099527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:06.100051 (+ 524us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.100398 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.100917 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":314,"threads_started":1}
I20260929 16:35:06.120222 19562 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:42791
I20260929 16:35:06.120303 19696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:42791 every 8 connection(s)
I20260929 16:35:06.121790 19562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:06.123741 19697 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40121
I20260929 16:35:06.124172 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.109601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.2:60097)
0929 16:35:06.110407 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.110431 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.110583 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.110970 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.110976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.111589 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:06.111880 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.111888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.113292 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.113298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.115112 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.115128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.116121 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.116129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.116254 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.116878 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:06.116901 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:06.118086 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:06.121677 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.121681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.121684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.122082 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.122505 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.122510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.122517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.122656 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.123349 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:06.123354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:06.123530 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.123638 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.123894 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":378,"threads_started":1}
I20260929 16:35:06.124526 19697 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:06.124974 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.109532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:60097 (local address 127.17.205.62:40121)
0929 16:35:06.109705 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.109711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.110531 (+ 820us) server_negotiation.cc:407] Connection header received
0929 16:35:06.110735 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.110741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.110821 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.110915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.112022 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.113146 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.115282 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.115914 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.116042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.118290 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.118297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.118302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:06.118335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.121568 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.122201 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.122208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.122212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.122363 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.122759 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.122764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.122767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.123200 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.124313 (+ 1113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.124591 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.124773 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:35:06.125204 19697 heartbeater.cc:511] Master 127.17.205.62:40121 requested a full tablet report, sending...
I20260929 16:35:06.126559 19351 ts_manager.cc:194] Registered new tserver with Master: db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:06.127447 19351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:59863
I20260929 16:35:06.128489 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19562
I20260929 16:35:06.128679 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:06.129197 18228 external_mini_cluster.cc:1428] Setting key c84a13d2ce6d25d7081a18ca3ea37cbd
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:06.166235 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40121
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:06.480471 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:06.480767 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:06.480907 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:06.488212 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:06.488338 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:06.488461 19701 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:06.488565 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:06.498375 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:06.499866 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:06.501296 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:06.514456 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:06.514508 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:06.514530 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:06.515580 19701 server_base.cc:1034] running on GCE node
I20260929 16:35:06.516206 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:06.517177 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:06.518410 19701 hybrid_clock.cc:648] HybridClock initialized: now 1790699706518339 us; error 53 us; skew 500 ppm
I20260929 16:35:06.518721 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:06.522141 19701 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:06.523808 19701 webserver.cc:460] Webserver started at http://127.17.205.3:38899/ using document root <none> and password file <none>
I20260929 16:35:06.524529 19701 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:06.524632 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:06.524937 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:06.527944 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "d14d577d1ab94a6c9570a8d19e94f47a"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "339e3d756bb08eecd0fcc99f75cebee1"
server_key_iv: "9825417ed519a3fbdf6e52debe153152"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.528739 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "d14d577d1ab94a6c9570a8d19e94f47a"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "339e3d756bb08eecd0fcc99f75cebee1"
server_key_iv: "9825417ed519a3fbdf6e52debe153152"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.534180 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:06.538172 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:06.539834 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:06.540033 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "d14d577d1ab94a6c9570a8d19e94f47a"
format_stamp: "Formatted at 2026-09-29 16:35:06 on dist-test-slave-3h01"
server_key: "339e3d756bb08eecd0fcc99f75cebee1"
server_key_iv: "9825417ed519a3fbdf6e52debe153152"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.540264 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:06.557796 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:06.562134 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:06.562390 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:06.565022 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:06.565107 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:06.565277 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:06.565354 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:06.587632 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.566562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:38703 (local address 127.17.205.62:40121)
0929 16:35:06.566801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.566808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.567116 (+ 308us) server_negotiation.cc:407] Connection header received
0929 16:35:06.569212 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.569219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.569305 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.569429 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.571588 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.572534 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.574800 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.575289 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.575422 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.580137 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.580141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.580145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:06.580179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.583656 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.584682 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.584687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.584691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.584874 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.585454 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.585461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.585465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.585869 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.586036 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.587013 (+ 977us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.587280 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:35:06.588789 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.566144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.3:38703)
0929 16:35:06.566953 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.567000 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.568895 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.569681 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.569695 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.570469 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:06.571393 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.571428 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.572690 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.572696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.574658 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.574671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.575552 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.575559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.575922 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.576780 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:06.576816 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:06.579900 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:06.583858 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.583869 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.583889 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.584537 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.584995 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.585000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.585003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.585287 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.586005 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:06.586015 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:06.586522 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.586870 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.587557 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":397,"threads_started":1}
I20260929 16:35:06.605871 19701 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:33217
I20260929 16:35:06.605942 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:33217 every 8 connection(s)
I20260929 16:35:06.607098 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:06.609337 19836 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40121
I20260929 16:35:06.609592 19836 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:06.610419 19836 heartbeater.cc:511] Master 127.17.205.62:40121 requested a full tablet report, sending...
I20260929 16:35:06.611716 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.596612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:49849 (local address 127.17.205.62:40121)
0929 16:35:06.596800 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.596807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.597587 (+ 780us) server_negotiation.cc:407] Connection header received
0929 16:35:06.597768 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.597773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.597844 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.597925 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.599080 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.600364 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.602693 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.603359 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.603492 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.605966 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.605973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.605978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:06.606019 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.608956 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.609806 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.609810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.609812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.609942 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.610312 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.610314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.610316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.610613 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.610717 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.611155 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.611436 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:35:06.611781 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.596748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.17.205.3:49849)
0929 16:35:06.597473 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.597489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.597637 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.597977 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.597982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.598610 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:06.598896 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.598906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.600595 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.600601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.602512 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.602532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.603764 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.603771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.603985 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.604578 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:06.604598 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:06.605694 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:06.609171 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.609177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.609181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.609702 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.610057 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:06.610062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:06.610066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:06.610218 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:06.610728 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:06.610733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:06.610929 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.611075 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.611402 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":407,"threads_started":1}
I20260929 16:35:06.612028 19351 ts_manager.cc:194] Registered new tserver with Master: d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3:33217)
I20260929 16:35:06.612712 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19701
I20260929 16:35:06.612890 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:06.613090 19351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:38703
I20260929 16:35:06.613334 18228 external_mini_cluster.cc:1428] Setting key 19b4175f419aa4c6fad6e3b55fe494cb
I20260929 16:35:06.617442 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 16:35:06.647330 19558 heartbeater.cc:503] Master 127.17.205.62:40121 was elected leader, sending a full tablet report...
Sep 29 16:35:06 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:06.662832 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.649263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.17.205.62:40121)
0929 16:35:06.649422 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.649429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.649805 (+ 376us) server_negotiation.cc:407] Connection header received
0929 16:35:06.649967 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.649972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.650043 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.650121 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.651444 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.652157 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.653383 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.653797 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.653919 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.657742 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.657748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.657751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:06.657787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.660279 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.660926 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.660931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.660934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.661054 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:06.661428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:06.661433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:06.661436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:06.661711 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.661946 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.662239 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.662447 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:35:06.666666 19351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:06.673420 19351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:06.708638 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.685224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.17.205.1:39505)
0929 16:35:06.700299 (+ 15075us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.700317 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.700343 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:35:06.700439 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.700447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.700729 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.701726 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:06.701924 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.703557 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.706396 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.707556 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.707679 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.707897 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.708275 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.708415 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14936,"thread_start_us":14368,"threads_started":1}
I20260929 16:35:06.709677 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.685007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:39505 (local address 127.0.0.1:47762)
0929 16:35:06.698114 (+ 13107us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.698178 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.699159 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.700976 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.700985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.701017 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:06.701492 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.701504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.703712 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.703718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.706227 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.706240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.707820 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.707827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.708004 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.708057 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.708950 (+ 893us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.709245 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12773,"mutex_wait_us":134,"thread_start_us":14974,"threads_started":1}
I20260929 16:35:06.709833 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.685109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:42791 (local address 127.0.0.1:34704)
0929 16:35:06.700213 (+ 15104us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.700231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.700350 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.701291 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.701296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.701324 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:06.701804 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.701816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.703797 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.703803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.706495 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.706508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.709179 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.709184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.709315 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.709338 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.709403 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.709492 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14988,"mutex_wait_us":26,"thread_start_us":13297,"threads_started":1}
I20260929 16:35:06.710472 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.685794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.17.205.2:42791)
0929 16:35:06.700041 (+ 14247us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.700050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.700323 (+ 273us) server_negotiation.cc:407] Connection header received
0929 16:35:06.700528 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.700536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.700793 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.700983 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:06.701980 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.703557 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.706648 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.707986 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.708110 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.710097 (+ 1987us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.710159 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.710283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14063,"thread_start_us":13923,"threads_started":1}
I20260929 16:35:06.713403 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.700785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.17.205.3:33217)
0929 16:35:06.705180 (+ 4395us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.705189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.705225 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:35:06.705326 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.705335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.705611 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.705798 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:06.706440 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.708573 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.710923 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.712350 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.712485 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.712707 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.712942 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.713103 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4227,"thread_start_us":4048,"threads_started":1}
I20260929 16:35:06.713408 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.700482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:33217 (local address 127.0.0.1:56760)
0929 16:35:06.704463 (+ 3981us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.704478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.704584 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.705799 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.705804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.705829 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:06.706237 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.706246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.708721 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.708726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.710774 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.710788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.712557 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.712563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.712744 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.712777 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.712890 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.713043 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3885,"mutex_wait_us":2,"thread_start_us":3728,"threads_started":1}
I20260929 16:35:06.713964 19488 tablet_service.cc:1467] Processing CreateTablet for tablet d590b2ac94654a039e71aeded12d2f03 (DEFAULT_TABLE table=test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.713976 19627 tablet_service.cc:1467] Processing CreateTablet for tablet d590b2ac94654a039e71aeded12d2f03 (DEFAULT_TABLE table=test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.715490 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590b2ac94654a039e71aeded12d2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.715487 19488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590b2ac94654a039e71aeded12d2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.717531 19766 tablet_service.cc:1467] Processing CreateTablet for tablet d590b2ac94654a039e71aeded12d2f03 (DEFAULT_TABLE table=test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.718979 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590b2ac94654a039e71aeded12d2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.727339 19855 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Bootstrap starting.
I20260929 16:35:06.729598 19856 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Bootstrap starting.
I20260929 16:35:06.731063 19855 tablet_bootstrap.cc:654] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.732422 19855 log.cc:824] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:06.733017 19856 tablet_bootstrap.cc:654] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.733268 19857 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Bootstrap starting.
I20260929 16:35:06.734593 19856 log.cc:824] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:06.736649 19857 tablet_bootstrap.cc:654] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.737242 19855 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: No bootstrap required, opened a new log
I20260929 16:35:06.737574 19855 ts_tablet_manager.cc:1397] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:35:06.737593 19857 log.cc:824] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:06.740340 19856 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: No bootstrap required, opened a new log
I20260929 16:35:06.740552 19857 tablet_bootstrap.cc:492] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: No bootstrap required, opened a new log
I20260929 16:35:06.740593 19856 ts_tablet_manager.cc:1397] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 16:35:06.740803 19857 ts_tablet_manager.cc:1397] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260929 16:35:06.747961 19855 raft_consensus.cc:348] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.747963 19856 raft_consensus.cc:348] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.748430 19856 raft_consensus.cc:374] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.748580 19856 raft_consensus.cc:729] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db58384b737a488487a76a480acd4a2c, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.748605 19855 raft_consensus.cc:374] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.748760 19855 raft_consensus.cc:729] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9422ad2bde40f6bb223aa8f9065407, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.749347 19856 consensus_queue.cc:260] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.749529 19855 consensus_queue.cc:260] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.750415 19697 heartbeater.cc:503] Master 127.17.205.62:40121 was elected leader, sending a full tablet report...
I20260929 16:35:06.751127 19856 ts_tablet_manager.cc:1428] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260929 16:35:06.750922 19857 raft_consensus.cc:348] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.751529 19857 raft_consensus.cc:374] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.751641 19857 raft_consensus.cc:729] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.752399 19857 consensus_queue.cc:260] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.753537 19836 heartbeater.cc:503] Master 127.17.205.62:40121 was elected leader, sending a full tablet report...
I20260929 16:35:06.754208 19857 ts_tablet_manager.cc:1428] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 16:35:06.758657 19855 ts_tablet_manager.cc:1428] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Time spent starting tablet: real 0.021s user 0.016s sys 0.003s
I20260929 16:35:06.773562 19863 raft_consensus.cc:482] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:06.773849 19863 raft_consensus.cc:504] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.775482 19863 leader_election.cc:288] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:06.781383 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.775777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:54583 (local address 127.17.205.1:39505)
0929 16:35:06.775958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.775964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.776247 (+ 283us) server_negotiation.cc:407] Connection header received
0929 16:35:06.776332 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.776337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.776426 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.776518 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:06.776849 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.777743 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.779548 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.780632 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.780742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.780795 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.781096 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.781178 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:35:06.781494 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.775692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:39505 (local address 127.17.205.3:54583)
0929 16:35:06.775991 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.776010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.776138 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.776544 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.776548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.776570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:06.776731 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.776739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.777879 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.777885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.779411 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.779423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.780761 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.780767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.780957 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.780991 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.781083 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.781216 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260929 16:35:06.782406 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.775980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:58009 (local address 127.17.205.2:42791)
0929 16:35:06.776147 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.776152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.776702 (+ 550us) server_negotiation.cc:407] Connection header received
0929 16:35:06.776888 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.776891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.776946 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.777009 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:06.777543 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.778468 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.780408 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.781400 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.781545 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.781609 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.782025 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.782159 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:35:06.782429 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.775879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:42791 (local address 127.17.205.3:58009)
0929 16:35:06.776605 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:06.776626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:06.776754 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:06.777046 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:06.777051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:06.777075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:06.777409 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.777421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.778597 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.778601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.780265 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:06.780279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.781748 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.781753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:06.781881 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.781903 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:35:06.781975 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:35:06.782097 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":423,"threads_started":1}
I20260929 16:35:06.782452 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590b2ac94654a039e71aeded12d2f03" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407" is_pre_election: true
I20260929 16:35:06.782998 19508 raft_consensus.cc:2454] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:06.783346 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590b2ac94654a039e71aeded12d2f03" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c" is_pre_election: true
I20260929 16:35:06.783702 19719 leader_election.cc:302] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9422ad2bde40f6bb223aa8f9065407, d14d577d1ab94a6c9570a8d19e94f47a; no voters:
I20260929 16:35:06.783854 19647 raft_consensus.cc:2454] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:06.784065 19863 raft_consensus.cc:2781] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:06.784170 19863 raft_consensus.cc:482] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:06.784250 19863 raft_consensus.cc:3037] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:06.786415 19863 raft_consensus.cc:504] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.786988 19863 leader_election.cc:288] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Requested vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:06.787497 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590b2ac94654a039e71aeded12d2f03" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c"
I20260929 16:35:06.787498 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590b2ac94654a039e71aeded12d2f03" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407"
I20260929 16:35:06.787712 19508 raft_consensus.cc:3037] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:06.787712 19647 raft_consensus.cc:3037] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:06.790766 19508 raft_consensus.cc:2454] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:06.790757 19647 raft_consensus.cc:2454] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:06.791159 19718 leader_election.cc:302] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d14d577d1ab94a6c9570a8d19e94f47a, db58384b737a488487a76a480acd4a2c; no voters:
I20260929 16:35:06.791494 19863 raft_consensus.cc:2781] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:06.791915 19863 raft_consensus.cc:686] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 LEADER]: Becoming Leader. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Running, Role: LEADER
I20260929 16:35:06.792356 19863 consensus_queue.cc:237] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.796984 19350 catalog_manager.cc:5495] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a reported cstate change: term changed from 0 to 1, leader changed from <none> to d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3). New cstate: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } health_report { overall_health: HEALTHY } } }
I20260929 16:35:06.826599 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.821233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.17.205.3:33217)
0929 16:35:06.821432 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.821437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.821456 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:35:06.821716 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.821720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.821807 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.821928 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:06.822216 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.822905 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.824533 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.825043 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.825137 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.825567 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:06.826103 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:06.826207 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.826325 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.826396 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:35:06.834394 19647 raft_consensus.cc:1261] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Refusing update from remote peer d14d577d1ab94a6c9570a8d19e94f47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:06.834394 19508 raft_consensus.cc:1261] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Refusing update from remote peer d14d577d1ab94a6c9570a8d19e94f47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:06.835309 19865 consensus_queue.cc:1035] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:06.835616 19863 consensus_queue.cc:1035] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:06.845086 19869 mvcc.cc:205] Tried to move back new op lower bound from 7334705999184576512 to 7334705999025217536. Current Snapshot: MvccSnapshot[applied={T|T < 7334705999184576512}]
I20260929 16:35:06.861099 19871 mvcc.cc:205] Tried to move back new op lower bound from 7334705999184576512 to 7334705999025217536. Current Snapshot: MvccSnapshot[applied={T|T < 7334705999184576512}]
I20260929 16:35:06.868182 19350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:06.868429 19350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:06.871939 19350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Sending DeleteTablet for 3 replicas of tablet d590b2ac94654a039e71aeded12d2f03
I20260929 16:35:06.873157 19627 tablet_service.cc:1514] Processing DeleteTablet for tablet d590b2ac94654a039e71aeded12d2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:06 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:34704
I20260929 16:35:06.873157 19488 tablet_service.cc:1514] Processing DeleteTablet for tablet d590b2ac94654a039e71aeded12d2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:06 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:47762
I20260929 16:35:06.874185 19766 tablet_service.cc:1514] Processing DeleteTablet for tablet d590b2ac94654a039e71aeded12d2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:06 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 16:35:06.874631 19350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:06.875288 19350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:06.877578 19880 tablet_replica.cc:330] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: stopping tablet replica
I20260929 16:35:06.878060 19880 raft_consensus.cc:2229] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:06.878736 19880 raft_consensus.cc:2258] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:06.884054 19626 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfee219dfe847f289e698ea5da72481 (DEFAULT_TABLE table=test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.884667 19626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfee219dfe847f289e698ea5da72481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.884608 19487 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfee219dfe847f289e698ea5da72481 (DEFAULT_TABLE table=test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.884841 19766 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfee219dfe847f289e698ea5da72481 (DEFAULT_TABLE table=test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:06.885218 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfee219dfe847f289e698ea5da72481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.885241 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfee219dfe847f289e698ea5da72481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.889681 19879 tablet_replica.cc:330] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: stopping tablet replica
I20260929 16:35:06.890156 19879 raft_consensus.cc:2229] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:06.891124 19878 tablet_replica.cc:330] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: stopping tablet replica
I20260929 16:35:06.891176 19856 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Bootstrap starting.
I20260929 16:35:06.891567 19878 raft_consensus.cc:2229] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:06.892117 19878 raft_consensus.cc:2258] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:06.892549 19879 raft_consensus.cc:2258] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:06.892803 19856 tablet_bootstrap.cc:654] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.894332 19878 ts_tablet_manager.cc:1905] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:06.894447 19879 ts_tablet_manager.cc:1905] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:06.895776 19857 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Bootstrap starting.
I20260929 16:35:06.897372 19857 tablet_bootstrap.cc:654] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.897431 19855 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Bootstrap starting.
I20260929 16:35:06.898888 19855 tablet_bootstrap.cc:654] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.901876 19879 ts_tablet_manager.cc:1918] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:06.901963 19880 ts_tablet_manager.cc:1905] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:06.902061 19879 log.cc:1196] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/d590b2ac94654a039e71aeded12d2f03
I20260929 16:35:06.902560 19879 ts_tablet_manager.cc:1939] T d590b2ac94654a039e71aeded12d2f03 P db58384b737a488487a76a480acd4a2c: Deleting consensus metadata
I20260929 16:35:06.904585 19336 catalog_manager.cc:4841] TS db58384b737a488487a76a480acd4a2c (127.17.205.2:42791): tablet d590b2ac94654a039e71aeded12d2f03 (table test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]) successfully deleted
I20260929 16:35:06.912546 19856 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: No bootstrap required, opened a new log
I20260929 16:35:06.912735 19856 ts_tablet_manager.cc:1397] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.009s
I20260929 16:35:06.913717 19878 ts_tablet_manager.cc:1918] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:06.913645 19856 raft_consensus.cc:348] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.913910 19878 log.cc:1196] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/d590b2ac94654a039e71aeded12d2f03
I20260929 16:35:06.913929 19856 raft_consensus.cc:374] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.913952 19857 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: No bootstrap required, opened a new log
I20260929 16:35:06.914042 19856 raft_consensus.cc:729] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db58384b737a488487a76a480acd4a2c, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.914270 19857 ts_tablet_manager.cc:1397] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20260929 16:35:06.914484 19878 ts_tablet_manager.cc:1939] T d590b2ac94654a039e71aeded12d2f03 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting consensus metadata
I20260929 16:35:06.914364 19856 consensus_queue.cc:260] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.915261 19857 raft_consensus.cc:348] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.915539 19857 raft_consensus.cc:374] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.915676 19857 raft_consensus.cc:729] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.916021 19857 consensus_queue.cc:260] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.916621 19857 ts_tablet_manager.cc:1428] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:35:06.917128 19337 catalog_manager.cc:4841] TS 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505): tablet d590b2ac94654a039e71aeded12d2f03 (table test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]) successfully deleted
I20260929 16:35:06.917256 19855 tablet_bootstrap.cc:492] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: No bootstrap required, opened a new log
I20260929 16:35:06.917384 19855 ts_tablet_manager.cc:1397] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20260929 16:35:06.918092 19855 raft_consensus.cc:348] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.918299 19855 raft_consensus.cc:374] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.918385 19855 raft_consensus.cc:729] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9422ad2bde40f6bb223aa8f9065407, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.918632 19855 consensus_queue.cc:260] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:06.918928 19880 ts_tablet_manager.cc:1918] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:06.919103 19880 log.cc:1196] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/d590b2ac94654a039e71aeded12d2f03
I20260929 16:35:06.919155 19855 ts_tablet_manager.cc:1428] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:35:06.919655 19880 ts_tablet_manager.cc:1939] T d590b2ac94654a039e71aeded12d2f03 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting consensus metadata
I20260929 16:35:06.919685 19856 ts_tablet_manager.cc:1428] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260929 16:35:06.921334 19337 catalog_manager.cc:4841] TS d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3:33217): tablet d590b2ac94654a039e71aeded12d2f03 (table test-table [id=51c0947712f54bd7b3bd53a07dc8eac9]) successfully deleted
W20260929 16:35:07.110591 19837 tablet.cc:2369] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:07.125327 19698 tablet.cc:2369] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:07.140862 19559 tablet.cc:2369] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:35:07.327134 19863 raft_consensus.cc:482] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:07.327468 19863 raft_consensus.cc:504] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.328109 19863 leader_election.cc:288] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:07.328634 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfee219dfe847f289e698ea5da72481" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407" is_pre_election: true
I20260929 16:35:07.328653 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfee219dfe847f289e698ea5da72481" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c" is_pre_election: true
I20260929 16:35:07.328853 19508 raft_consensus.cc:2454] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:07.328855 19647 raft_consensus.cc:2454] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:07.329267 19719 leader_election.cc:302] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b9422ad2bde40f6bb223aa8f9065407, d14d577d1ab94a6c9570a8d19e94f47a, db58384b737a488487a76a480acd4a2c; no voters:
I20260929 16:35:07.329514 19863 raft_consensus.cc:2781] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:07.329615 19863 raft_consensus.cc:482] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:07.329698 19863 raft_consensus.cc:3037] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.331694 19863 raft_consensus.cc:504] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.332213 19863 leader_election.cc:288] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Requested vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:07.332723 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfee219dfe847f289e698ea5da72481" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c"
I20260929 16:35:07.332736 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfee219dfe847f289e698ea5da72481" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407"
I20260929 16:35:07.332926 19647 raft_consensus.cc:3037] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.332918 19508 raft_consensus.cc:3037] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.334836 19508 raft_consensus.cc:2454] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:07.335012 19647 raft_consensus.cc:2454] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:07.335330 19719 leader_election.cc:302] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9422ad2bde40f6bb223aa8f9065407, d14d577d1ab94a6c9570a8d19e94f47a; no voters:
I20260929 16:35:07.335642 19863 raft_consensus.cc:2781] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:07.335767 19863 raft_consensus.cc:686] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 LEADER]: Becoming Leader. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Running, Role: LEADER
I20260929 16:35:07.336037 19863 consensus_queue.cc:237] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.338394 19349 catalog_manager.cc:5495] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a reported cstate change: term changed from 0 to 1, leader changed from <none> to d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3). New cstate: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } health_report { overall_health: HEALTHY } } }
Sep 29 16:35:07 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:07.382565 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.369712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40121 (local address 127.0.0.1:54428)
0929 16:35:07.370426 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:07.370444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:07.370584 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:07.370879 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:07.370884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:07.371610 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:07.371846 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.371856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.372913 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.372916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.373843 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.373854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.374531 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.374536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.374706 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.375343 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:07.375360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:07.377413 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:07.380181 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:07.380195 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:07.380203 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:07.380614 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:07.380943 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:07.380948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:07.380952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:07.381079 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:07.381555 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:07.381563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:07.381721 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 16:35:07.381868 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:35:07.382142 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":367,"threads_started":1}
I20260929 16:35:07.382638 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.369706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.17.205.62:40121)
0929 16:35:07.370395 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.370402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.370519 (+ 117us) server_negotiation.cc:407] Connection header received
0929 16:35:07.370687 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.370692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.370747 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.370830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.372008 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.372778 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.373965 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.374386 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.374486 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.377595 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.377600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.377603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:07.377631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.380064 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.380724 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.380730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.380733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.380846 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.381170 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.381175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.381178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.381438 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.381559 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.381968 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.382285 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":495,"thread_start_us":318,"threads_started":1}
I20260929 16:35:07.387517 19349 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:54428:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:35:07.388500 19349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:07.394811 19627 tablet_service.cc:1467] Processing CreateTablet for tablet 5944b298539843f3abe0ef48ccd98cbd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1063fdcea59e485982b24d6ba8c6882e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:07.394810 19487 tablet_service.cc:1467] Processing CreateTablet for tablet 5944b298539843f3abe0ef48ccd98cbd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1063fdcea59e485982b24d6ba8c6882e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:07.395352 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5944b298539843f3abe0ef48ccd98cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:07.395357 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5944b298539843f3abe0ef48ccd98cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:07.395196 19766 tablet_service.cc:1467] Processing CreateTablet for tablet 5944b298539843f3abe0ef48ccd98cbd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1063fdcea59e485982b24d6ba8c6882e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:07.395727 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5944b298539843f3abe0ef48ccd98cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:07.400695 19856 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: Bootstrap starting.
I20260929 16:35:07.402051 19855 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: Bootstrap starting.
I20260929 16:35:07.402065 19856 tablet_bootstrap.cc:654] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:07.402542 19857 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: Bootstrap starting.
I20260929 16:35:07.403599 19855 tablet_bootstrap.cc:654] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:07.403990 19857 tablet_bootstrap.cc:654] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:07.404680 19856 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: No bootstrap required, opened a new log
I20260929 16:35:07.404827 19856 ts_tablet_manager.cc:1397] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 16:35:07.405601 19855 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: No bootstrap required, opened a new log
I20260929 16:35:07.405568 19856 raft_consensus.cc:348] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.405764 19855 ts_tablet_manager.cc:1397] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:07.405782 19856 raft_consensus.cc:374] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:07.405848 19856 raft_consensus.cc:729] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db58384b737a488487a76a480acd4a2c, State: Initialized, Role: FOLLOWER
I20260929 16:35:07.406092 19856 consensus_queue.cc:260] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.406394 19855 raft_consensus.cc:348] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.406595 19855 raft_consensus.cc:374] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:07.406625 19856 ts_tablet_manager.cc:1428] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:35:07.406663 19855 raft_consensus.cc:729] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9422ad2bde40f6bb223aa8f9065407, State: Initialized, Role: FOLLOWER
I20260929 16:35:07.406533 19861 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.406813 19861 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.406884 19855 consensus_queue.cc:260] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.407325 19862 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.407606 19862 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.408164 19855 ts_tablet_manager.cc:1428] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:07.409379 19857 tablet_bootstrap.cc:492] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: No bootstrap required, opened a new log
I20260929 16:35:07.409524 19857 ts_tablet_manager.cc:1397] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 16:35:07.410360 19857 raft_consensus.cc:348] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.410660 19857 raft_consensus.cc:374] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:07.410751 19857 raft_consensus.cc:729] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Initialized, Role: FOLLOWER
I20260929 16:35:07.410977 19857 consensus_queue.cc:260] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.411453 19857 ts_tablet_manager.cc:1428] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:35:07.411369 19863 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.411674 19863 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.725557 19863 raft_consensus.cc:482] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:07.725780 19863 raft_consensus.cc:504] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.726366 19863 leader_election.cc:288] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:07.726945 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5944b298539843f3abe0ef48ccd98cbd" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c" is_pre_election: true
I20260929 16:35:07.726948 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5944b298539843f3abe0ef48ccd98cbd" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407" is_pre_election: true
I20260929 16:35:07.727155 19508 raft_consensus.cc:2454] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:07.727186 19647 raft_consensus.cc:2454] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 0.
I20260929 16:35:07.727586 19719 leader_election.cc:302] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9422ad2bde40f6bb223aa8f9065407, d14d577d1ab94a6c9570a8d19e94f47a; no voters:
I20260929 16:35:07.727851 19863 raft_consensus.cc:2781] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:07.727932 19863 raft_consensus.cc:482] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:07.728005 19863 raft_consensus.cc:3037] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.729905 19863 raft_consensus.cc:504] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.730406 19863 leader_election.cc:288] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Requested vote from peers 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505), db58384b737a488487a76a480acd4a2c (127.17.205.2:42791)
I20260929 16:35:07.730815 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5944b298539843f3abe0ef48ccd98cbd" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9422ad2bde40f6bb223aa8f9065407"
I20260929 16:35:07.730909 19647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5944b298539843f3abe0ef48ccd98cbd" candidate_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db58384b737a488487a76a480acd4a2c"
I20260929 16:35:07.731026 19508 raft_consensus.cc:3037] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.731127 19647 raft_consensus.cc:3037] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.733004 19647 raft_consensus.cc:2454] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:07.733006 19508 raft_consensus.cc:2454] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14d577d1ab94a6c9570a8d19e94f47a in term 1.
I20260929 16:35:07.733417 19718 leader_election.cc:302] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b9422ad2bde40f6bb223aa8f9065407, d14d577d1ab94a6c9570a8d19e94f47a, db58384b737a488487a76a480acd4a2c; no voters:
I20260929 16:35:07.733678 19863 raft_consensus.cc:2781] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:07.733868 19863 raft_consensus.cc:686] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [term 1 LEADER]: Becoming Leader. State: Replica: d14d577d1ab94a6c9570a8d19e94f47a, State: Running, Role: LEADER
I20260929 16:35:07.734127 19863 consensus_queue.cc:237] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } }
I20260929 16:35:07.735702 19874 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: TxnStatusTablet state changed. Reason: New leader d14d577d1ab94a6c9570a8d19e94f47a. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.736028 19874 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:07.736572 19896 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:35:07.736792 19896 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:35:07.737200 19351 catalog_manager.cc:5495] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a reported cstate change: term changed from 0 to 1, leader changed from <none> to d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3). New cstate: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } health_report { overall_health: HEALTHY } } }
I20260929 16:35:07.778177 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.772076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:33217 (local address 127.0.0.1:56770)
0929 16:35:07.772344 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:07.772364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:07.772506 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:07.773408 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:07.773411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:07.773422 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:07.773585 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.773591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.775062 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.775065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.776110 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.776123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.777100 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.777105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.777266 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.777352 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:07.777757 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:07.777768 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:35:07.777824 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 16:35:07.777944 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:35:07.778342 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.772230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.17.205.3:33217)
0929 16:35:07.773074 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.773079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.773098 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:35:07.773165 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.773170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.773258 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.773380 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:07.773742 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.774948 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.776245 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.776951 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.777071 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.777535 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:07.777673 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:07.777762 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.777884 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.777965 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":545,"threads_started":1}
I20260929 16:35:07.782469 19647 raft_consensus.cc:1261] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Refusing update from remote peer d14d577d1ab94a6c9570a8d19e94f47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:07.782622 19508 raft_consensus.cc:1261] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Refusing update from remote peer d14d577d1ab94a6c9570a8d19e94f47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:07.783092 19863 consensus_queue.cc:1035] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:07.783442 19874 consensus_queue.cc:1035] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:07.792526 19862 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: TxnStatusTablet state changed. Reason: New leader d14d577d1ab94a6c9570a8d19e94f47a. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.792814 19862 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.793435 19861 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: TxnStatusTablet state changed. Reason: New leader d14d577d1ab94a6c9570a8d19e94f47a. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.793730 19861 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.799346 19900 mvcc.cc:205] Tried to move back new op lower bound from 7334706003071111168 to 7334706002881232896. Current Snapshot: MvccSnapshot[applied={T|T < 7334706003071111168}]
I20260929 16:35:07.806903 19863 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.807256 19863 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:07.813253 19863 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.813356 19862 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.813555 19863 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P d14d577d1ab94a6c9570a8d19e94f47a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:07.813570 19862 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P 3b9422ad2bde40f6bb223aa8f9065407: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.815248 19863 consensus_queue.cc:1035] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 16:35:07.815879 19861 tablet_replica.cc:439] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9422ad2bde40f6bb223aa8f9065407" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 39505 } } peers { permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 } } peers { permanent_uuid: "d14d577d1ab94a6c9570a8d19e94f47a" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33217 } } }
I20260929 16:35:07.816092 19861 tablet_replica.cc:442] T 5944b298539843f3abe0ef48ccd98cbd P db58384b737a488487a76a480acd4a2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:07.820643 19863 consensus_queue.cc:1035] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db58384b737a488487a76a480acd4a2c" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:07.852516 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.847410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:36751 (local address 127.17.205.3:33217)
0929 16:35:07.847557 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.847567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.847931 (+ 364us) server_negotiation.cc:407] Connection header received
0929 16:35:07.848115 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.848120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.848189 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.848261 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:07.848749 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.849491 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.850888 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.851356 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.851467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.851859 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:07.851963 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:07.852034 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.852146 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.852215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":29}
I20260929 16:35:07.852605 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.847341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:33217 (local address 127.17.205.3:36751)
0929 16:35:07.847838 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:07.847856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:07.847987 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:07.848275 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:07.848279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:07.848289 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:07.848611 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.848620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.849622 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.849628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.850755 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:07.850767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.851504 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.851510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:07.851635 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.851692 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:07.852044 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:07.852053 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:35:07.852121 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:35:07.852252 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260929 16:35:07.911561 19351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:07.911823 19351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:07.914969 19351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 278d26fe848848e18912ba47527a01ae: Sending DeleteTablet for 3 replicas of tablet 9bfee219dfe847f289e698ea5da72481
I20260929 16:35:07.915772 19487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfee219dfe847f289e698ea5da72481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:47762
I20260929 16:35:07.915835 19627 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfee219dfe847f289e698ea5da72481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:34704
I20260929 16:35:07.916047 19765 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfee219dfe847f289e698ea5da72481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 16:35:07.929560 19916 tablet_replica.cc:330] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: stopping tablet replica
I20260929 16:35:07.929849 19916 raft_consensus.cc:2229] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:07.930106 19916 raft_consensus.cc:2258] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:07.933295 19916 ts_tablet_manager.cc:1905] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:07.934309 19917 tablet_replica.cc:330] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: stopping tablet replica
I20260929 16:35:07.934640 19917 raft_consensus.cc:2229] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:07.935286 19918 tablet_replica.cc:330] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: stopping tablet replica
I20260929 16:35:07.935475 19918 raft_consensus.cc:2229] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:07.935850 19918 raft_consensus.cc:2258] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:07.938190 19918 ts_tablet_manager.cc:1905] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:07.938809 19917 raft_consensus.cc:2258] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:07.939927 19917 ts_tablet_manager.cc:1905] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:07.945163 19918 ts_tablet_manager.cc:1918] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:07.945297 19918 log.cc:1196] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/9bfee219dfe847f289e698ea5da72481
I20260929 16:35:07.945737 19918 ts_tablet_manager.cc:1939] T 9bfee219dfe847f289e698ea5da72481 P d14d577d1ab94a6c9570a8d19e94f47a: Deleting consensus metadata
I20260929 16:35:07.946808 19337 catalog_manager.cc:4841] TS d14d577d1ab94a6c9570a8d19e94f47a (127.17.205.3:33217): tablet 9bfee219dfe847f289e698ea5da72481 (table test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]) successfully deleted
I20260929 16:35:07.949239 19916 ts_tablet_manager.cc:1918] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:07.949384 19916 log.cc:1196] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/9bfee219dfe847f289e698ea5da72481
Sep 29 16:35:07 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:07.949810 19916 ts_tablet_manager.cc:1939] T 9bfee219dfe847f289e698ea5da72481 P 3b9422ad2bde40f6bb223aa8f9065407: Deleting consensus metadata
I20260929 16:35:07.950407 19917 ts_tablet_manager.cc:1918] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:07.950541 19917 log.cc:1196] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/9bfee219dfe847f289e698ea5da72481
I20260929 16:35:07.950968 19917 ts_tablet_manager.cc:1939] T 9bfee219dfe847f289e698ea5da72481 P db58384b737a488487a76a480acd4a2c: Deleting consensus metadata
I20260929 16:35:07.950949 19337 catalog_manager.cc:4841] TS 3b9422ad2bde40f6bb223aa8f9065407 (127.17.205.1:39505): tablet 9bfee219dfe847f289e698ea5da72481 (table test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]) successfully deleted
I20260929 16:35:07.952359 19336 catalog_manager.cc:4841] TS db58384b737a488487a76a480acd4a2c (127.17.205.2:42791): tablet 9bfee219dfe847f289e698ea5da72481 (table test-table [id=048c3d62f1b643cbbe64ecaefa8a6ce9]) successfully deleted
I20260929 16:35:07.957219 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.927584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.17.205.1:39505)
0929 16:35:07.933615 (+ 6031us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.933623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.934489 (+ 866us) server_negotiation.cc:407] Connection header received
0929 16:35:07.934564 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.934570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.934664 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.934761 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.937520 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.938656 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.940724 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.941598 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.941695 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.951073 (+ 9378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.951094 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.951116 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:07.951159 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.954493 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.955250 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.955267 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.955269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.955401 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.955770 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.955773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.955775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.956109 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.956396 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.956681 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.956899 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5888,"thread_start_us":6030,"threads_started":1}
W20260929 16:35:07.957942 19553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47770
I20260929 16:35:07.981096 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.960827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.17.205.62:40121)
0929 16:35:07.967644 (+ 6817us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.967649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.967668 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:35:07.967722 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.967727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.967836 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.967933 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.969717 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.970574 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.972393 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.972849 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.972960 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.975842 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.975848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.975851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:07.975897 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.978414 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.979170 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.979174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.979176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.979348 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.979794 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.979798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.979800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.980105 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.980322 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.980620 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.980797 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6706,"thread_start_us":6559,"threads_started":1}
W20260929 16:35:07.981633 19351 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54434
I20260929 16:35:07.983377 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19422
I20260929 16:35:07.996073 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.984444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.17.205.1:39505)
0929 16:35:07.984643 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.984650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.984769 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:35:07.984946 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.984951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.985023 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.985103 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.986160 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.986825 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.988072 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:07.988471 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:07.988580 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:07.990823 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.990828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.990831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:07.990866 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.993455 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.994124 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.994129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.994132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.994268 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:07.994680 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:07.994685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:07.994689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:07.994988 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.995242 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.995547 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.995765 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 16:35:07.996603 19553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47776
W20260929 16:35:07.997123 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19422: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:07.997284 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19422
I20260929 16:35:08.017629 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19562
Sep 29 16:35:08 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:08.033146 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:08.018422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.17.205.2:42791)
0929 16:35:08.019184 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:08.019189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:08.019204 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:35:08.019295 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:08.019299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:08.019366 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:08.019464 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:08.020658 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.021413 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.022658 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.023085 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.023193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:08.027380 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:08.027397 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:08.027414 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:08.027451 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:08.030300 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.031013 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.031023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.031025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.031189 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.031602 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.031607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.031610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.031982 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:08.032249 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:08.032570 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:35:08.032811 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":527,"threads_started":1}
W20260929 16:35:08.033828 19692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34718
W20260929 16:35:08.034458 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19562: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:08.034613 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19562
I20260929 16:35:08.055392 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19701
Sep 29 16:35:08 dist-test-slave-3h01 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699706, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:08.076155 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:08.056238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.17.205.3:33217)
0929 16:35:08.056438 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:08.056444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:08.056556 (+ 112us) server_negotiation.cc:407] Connection header received
0929 16:35:08.056759 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:08.056766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:08.056850 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:08.056965 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:08.058587 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.059995 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.061866 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.062577 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.062717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:08.068422 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:08.068449 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:08.068470 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:08.068523 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:08.072970 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.073724 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.073735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.073739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.073942 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.074425 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.074431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.074435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.074907 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:08.075279 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:08.075531 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:35:08.075765 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:35:08.076905 19831 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56782
W20260929 16:35:08.077687 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19701: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:08.077863 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19701
I20260929 16:35:08.097467 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19319
I20260929 16:35:08.098613 19391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:08.209920 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19319
2026-09-29T16:35:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4725 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:08 dist-test-slave-3h01 krb5kdc[19930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:08 dist-test-slave-3h01 krb5kdc[19930](info): set up 2 sockets
Sep 29 16:35:08 dist-test-slave-3h01 krb5kdc[19930](info): commencing operation
krb5kdc: starting...
W20260929 16:35:09.207132 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:09 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:09.382755 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34533
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:34533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:09.698020 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:09.698338 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:09.698467 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:09.705651 19943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:09.705778 19943 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:35:09.705857 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:09.705941 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:09.706007 19943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:09.706089 19943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:09.715811 19943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:34533
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:09.717108 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:09.718430 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:09.729784 19948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:09.729777 19951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:09.729805 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:09.730562 19943 server_base.cc:1034] running on GCE node
I20260929 16:35:09.731384 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:09.732766 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:09.733974 19943 hybrid_clock.cc:648] HybridClock initialized: now 1790699709733900 us; error 54 us; skew 500 ppm
I20260929 16:35:09.734282 19943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:09 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699709, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:09.737170 19943 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:09.738564 19943 webserver.cc:460] Webserver started at http://127.17.205.62:36133/ using document root <none> and password file <none>
I20260929 16:35:09.739295 19943 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:09.739408 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:09.739702 19943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:09.742614 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "a49336ef52e34b09ba1a350a12101527"
format_stamp: "Formatted at 2026-09-29 16:35:09 on dist-test-slave-3h01"
server_key: "bdaeecca68c6710b7f89eaed5c2d6165"
server_key_iv: "3838d72ba9469bf2aa964325657a779c"
server_key_version: "encryptionkey@0"
I20260929 16:35:09.743422 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "a49336ef52e34b09ba1a350a12101527"
format_stamp: "Formatted at 2026-09-29 16:35:09 on dist-test-slave-3h01"
server_key: "bdaeecca68c6710b7f89eaed5c2d6165"
server_key_iv: "3838d72ba9469bf2aa964325657a779c"
server_key_version: "encryptionkey@0"
I20260929 16:35:09.748486 19943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:09.752077 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:09.753492 19943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:09.753649 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a49336ef52e34b09ba1a350a12101527"
format_stamp: "Formatted at 2026-09-29 16:35:09 on dist-test-slave-3h01"
server_key: "bdaeecca68c6710b7f89eaed5c2d6165"
server_key_iv: "3838d72ba9469bf2aa964325657a779c"
server_key_version: "encryptionkey@0"
I20260929 16:35:09.753841 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:09.772665 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:09.777017 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:09.777279 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:09.796193 19943 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:34533
I20260929 16:35:09.796216 20019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:34533 every 8 connection(s)
I20260929 16:35:09.797523 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:09.798983 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 19943
I20260929 16:35:09.799165 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:09.799795 18228 external_mini_cluster.cc:1428] Setting key 9784c6e042ec5b2155a3c0c776074b4f
I20260929 16:35:09.801869 20020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:35:09 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:09.814872 20020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Bootstrap starting.
I20260929 16:35:09.818821 20020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:09.820078 20020 log.cc:824] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:09.822950 20020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: No bootstrap required, opened a new log
I20260929 16:35:09.823428 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:09.801818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.17.205.62:34533)
0929 16:35:09.802626 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:09.802642 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:09.802692 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:35:09.804794 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:09.804824 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:09.805402 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:09.805875 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:09.807105 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:09.809147 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:09.810506 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:09.811546 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:09.811727 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:09.815732 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:09.815742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:09.815762 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:09.815809 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:09.819441 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:09.820147 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:09.820155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:09.820163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:09.820361 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:09.820698 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:09.820703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:09.820705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:09.821259 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:09.821553 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:09.821951 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 16:35:09.822346 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":320,"threads_started":1}
I20260929 16:35:09.830816 20020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } }
I20260929 16:35:09.831261 20020 raft_consensus.cc:374] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:09.831377 20020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49336ef52e34b09ba1a350a12101527, State: Initialized, Role: FOLLOWER
I20260929 16:35:09.831971 20020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } }
I20260929 16:35:09.832214 20020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:09.832376 20020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:09.832566 20020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:09.834673 20020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } }
I20260929 16:35:09.835170 20020 leader_election.cc:302] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49336ef52e34b09ba1a350a12101527; no voters:
I20260929 16:35:09.835721 20020 leader_election.cc:288] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:09.835834 20025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:09.836470 20025 raft_consensus.cc:686] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [term 1 LEADER]: Becoming Leader. State: Replica: a49336ef52e34b09ba1a350a12101527, State: Running, Role: LEADER
I20260929 16:35:09.836953 20025 consensus_queue.cc:237] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } }
I20260929 16:35:09.837775 20020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:09.839926 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a49336ef52e34b09ba1a350a12101527. Latest consensus state: current_term: 1 leader_uuid: "a49336ef52e34b09ba1a350a12101527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } } }
I20260929 16:35:09.840608 20026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a49336ef52e34b09ba1a350a12101527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49336ef52e34b09ba1a350a12101527" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34533 } } }
I20260929 16:35:09.840791 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:09.840873 20026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:09.843036 20033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:09.848137 20033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:09.848834 20042 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:35:09.857030 20033 catalog_manager.cc:1347] Generated new cluster ID: 2b248c79167e44d0b32418af9aefa306
I20260929 16:35:09.857144 20033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:09.874980 20033 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:09.876557 20033 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:09.886025 20033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Generated new TSK 0
I20260929 16:35:09.886912 20033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:09.929958 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:34533
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:10.255347 20046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:10.255656 20046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:10.255796 20046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:10.263134 20046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:10.263301 20046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:10.263434 20046 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:35:10.263502 20046 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:10.263605 20046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:10.273619 20046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:34533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:10.275027 20046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:10.276398 20046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:10.289345 20052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.289322 20051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.289527 20054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:10.289737 20046 server_base.cc:1034] running on GCE node
I20260929 16:35:10.290645 20046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:10.291632 20046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:10.292845 20046 hybrid_clock.cc:648] HybridClock initialized: now 1790699710292773 us; error 56 us; skew 500 ppm
I20260929 16:35:10.293166 20046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:10 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:10.296310 20046 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:10.297736 20046 webserver.cc:460] Webserver started at http://127.17.205.1:41133/ using document root <none> and password file <none>
I20260929 16:35:10.298403 20046 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:10.298508 20046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:10.298802 20046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:10.301692 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "265fcb7d1064424a8a8e484d40ccef42"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "ac2b818af148ecce6ea2764be5c29fc7"
server_key_iv: "f7ede49d3c098e3c98f18b3fcae8b177"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.302420 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "265fcb7d1064424a8a8e484d40ccef42"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "ac2b818af148ecce6ea2764be5c29fc7"
server_key_iv: "f7ede49d3c098e3c98f18b3fcae8b177"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.307672 20046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:35:10.311110 20061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:10.312484 20046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:10.312633 20046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "265fcb7d1064424a8a8e484d40ccef42"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "ac2b818af148ecce6ea2764be5c29fc7"
server_key_iv: "f7ede49d3c098e3c98f18b3fcae8b177"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.312825 20046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:10.330655 20046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:10.334712 20046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:10.334967 20046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:10.337611 20046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:10.337738 20046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:10.337855 20046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:10.337929 20046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:10 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:10.362037 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.338480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:58149 (local address 127.17.205.62:34533)
0929 16:35:10.339026 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.339033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.340299 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 16:35:10.343056 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.343063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.343151 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.343264 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.346183 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.347065 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.349823 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.350312 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.350445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.355550 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.355556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.355559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:10.355594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.358605 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.359552 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.359559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.359563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.359729 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.360151 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.360156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.360160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.360528 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.360721 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.361355 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.361595 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":240,"threads_started":1}
I20260929 16:35:10.362823 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.339185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.1:58149)
0929 16:35:10.340095 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:10.340146 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:10.342679 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:10.343905 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:10.343919 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:10.344904 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:10.345932 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.345992 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.347280 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.347286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.349653 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.349674 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.350564 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.350572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.350823 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.351660 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:10.351694 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:10.355275 (+ 3581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:10.358817 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.358830 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.358855 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.359407 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.359837 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.359840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.359842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.360052 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.360636 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:10.360642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:10.361014 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 16:35:10.361235 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 16:35:10.361634 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":314,"threads_started":1}
I20260929 16:35:10.381822 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.368325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:58055 (local address 127.17.205.62:34533)
0929 16:35:10.368475 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.368479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.369032 (+ 553us) server_negotiation.cc:407] Connection header received
0929 16:35:10.369288 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.369294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.369379 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.369490 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.370407 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.371584 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.372913 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.373605 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.373728 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.375424 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.375429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.375432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:10.375463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.378740 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.379715 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.379722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.379726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.379905 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.380349 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.380353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.380355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.380657 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.380782 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.381354 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.381552 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:35:10.382042 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.368403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.1:58055)
0929 16:35:10.368936 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:10.368951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:10.369089 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:10.369536 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:10.369541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:10.370000 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:10.370279 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.370288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.371747 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.371751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.372783 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.372795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.373758 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.373779 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.373899 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.374520 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:10.374536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:10.375263 (+ 727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:10.378900 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.378904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.378907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.379539 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.380072 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.380080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.380084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.380238 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.380781 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:10.380786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:10.380994 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 16:35:10.381187 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 16:35:10.381550 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260929 16:35:10.390774 20046 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:33137
I20260929 16:35:10.390810 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:33137 every 8 connection(s)
I20260929 16:35:10.392030 20046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:10.394112 20182 heartbeater.cc:347] Connected to a master server at 127.17.205.62:34533
I20260929 16:35:10.394356 20182 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:10.395061 20182 heartbeater.cc:511] Master 127.17.205.62:34533 requested a full tablet report, sending...
I20260929 16:35:10.397163 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20046
I20260929 16:35:10.397181 19975 ts_manager.cc:194] Registered new tserver with Master: 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137)
I20260929 16:35:10.397336 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:10.397907 18228 external_mini_cluster.cc:1428] Setting key 8601aba0db62c6e444885c61cfe8b5ed
I20260929 16:35:10.398929 19975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:58149
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:10.438045 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:34533
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:10.747563 20186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:10.747851 20186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:10.747987 20186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:10.755256 20186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:10.755364 20186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:10.755484 20186 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:35:10.755546 20186 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:10.755645 20186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:10.765717 20186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:34533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:10.767091 20186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:10.768477 20186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:10.781368 20194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.781412 20192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.781677 20191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:10.782317 20186 server_base.cc:1034] running on GCE node
I20260929 16:35:10.782927 20186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:10.783988 20186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:10.785178 20186 hybrid_clock.cc:648] HybridClock initialized: now 1790699710785094 us; error 88 us; skew 500 ppm
I20260929 16:35:10.785487 20186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:10 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:10.788543 20186 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:10.790019 20186 webserver.cc:460] Webserver started at http://127.17.205.2:46641/ using document root <none> and password file <none>
I20260929 16:35:10.790679 20186 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:10.790773 20186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:10.791039 20186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:10.793990 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "7e24c6990b235c4ae2ed25a73b1955b9"
server_key_iv: "496d57ab74b7f2ab1b6bfabb78f9b63d"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.794718 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "7e24c6990b235c4ae2ed25a73b1955b9"
server_key_iv: "496d57ab74b7f2ab1b6bfabb78f9b63d"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.801442 20186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260929 16:35:10.806102 20201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:10.807873 20186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:10.808066 20186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-3h01"
server_key: "7e24c6990b235c4ae2ed25a73b1955b9"
server_key_iv: "496d57ab74b7f2ab1b6bfabb78f9b63d"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.808279 20186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:10.821467 20186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:10.825301 20186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:10.825526 20186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:10.828071 20186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:10.828162 20186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:10.828282 20186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:10.828351 20186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:10 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:10.850445 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.828611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:59129 (local address 127.17.205.62:34533)
0929 16:35:10.828812 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.828818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.830251 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 16:35:10.832373 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.832379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.832498 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.832576 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.835158 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.836314 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.838487 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.839108 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.839254 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.844090 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.844097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.844103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:10.844142 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.846976 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.847938 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.847945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.847949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.848129 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.848551 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.848556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.848560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.848926 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.849079 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.849788 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.850030 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:35:10.851269 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.829238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.2:59129)
0929 16:35:10.830065 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:10.830118 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:10.832022 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:10.833086 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:10.833102 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:10.834097 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:10.834914 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.834941 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.836485 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.836491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.838347 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.838366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.839414 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.839422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.839747 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.840603 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:10.840638 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:10.843802 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:10.847105 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.847119 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.847141 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.847799 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.848249 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.848253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.848255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.848446 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.849049 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:10.849055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:10.849432 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 16:35:10.849656 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 16:35:10.850052 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260929 16:35:10.869892 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.856575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:34981 (local address 127.17.205.62:34533)
0929 16:35:10.856758 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.856765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.857291 (+ 526us) server_negotiation.cc:407] Connection header received
0929 16:35:10.857548 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.857553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.857633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.857707 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.858699 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.859463 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.861537 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.862082 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.862203 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.864214 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.864220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.864226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:10.864266 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.867253 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.868102 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.868108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.868112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.868244 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:10.868565 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:10.868570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:10.868574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:10.868905 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.869045 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.869494 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.869668 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:35:10.870285 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.856597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.2:34981)
0929 16:35:10.857180 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:10.857209 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:10.857354 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:10.857898 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:10.857901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:10.858307 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:10.858572 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.858580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.859641 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.859646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.861345 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:10.861364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.862459 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.862466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:10.862597 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.863294 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:10.863316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:10.864024 (+ 708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:10.867415 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.867423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.867428 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.867990 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.868342 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:10.868347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:10.868351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:10.868471 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:10.869040 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:10.869046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:10.869260 (+ 214us) client_negotiation.cc:769] Sending connection context
0929 16:35:10.869431 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:35:10.869791 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
I20260929 16:35:10.877511 20186 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:42855
I20260929 16:35:10.877542 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:42855 every 8 connection(s)
I20260929 16:35:10.878682 20186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:10.880828 20321 heartbeater.cc:347] Connected to a master server at 127.17.205.62:34533
I20260929 16:35:10.881032 20321 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:10.881635 20321 heartbeater.cc:511] Master 127.17.205.62:34533 requested a full tablet report, sending...
I20260929 16:35:10.882951 19975 ts_manager.cc:194] Registered new tserver with Master: 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855)
I20260929 16:35:10.883757 19975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:59129
I20260929 16:35:10.885684 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20186
I20260929 16:35:10.885875 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:10.886322 18228 external_mini_cluster.cc:1428] Setting key 540eecb321097660c8c70f8d11337f93
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:10.922948 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:34533
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:11.245568 20325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:11.245860 20325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:11.246001 20325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:11.253173 20325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:11.253295 20325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:11.253417 20325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:35:11.253484 20325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:35:11.253582 20325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:11.263561 20325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:34533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:11.264961 20325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:11.266353 20325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:11.279152 20331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:11.279174 20330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:11.279235 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:11.280054 20325 server_base.cc:1034] running on GCE node
I20260929 16:35:11.280648 20325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:11.281565 20325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:11.282765 20325 hybrid_clock.cc:648] HybridClock initialized: now 1790699711282699 us; error 53 us; skew 500 ppm
I20260929 16:35:11.283105 20325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:11 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:11.286166 20325 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:11.287709 20325 webserver.cc:460] Webserver started at http://127.17.205.3:39685/ using document root <none> and password file <none>
I20260929 16:35:11.288393 20325 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:11.288496 20325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:11.288786 20325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:11.291853 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "3782174e37fc413ca42c3c7753b76678"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-3h01"
server_key: "0520f053c9b17948c84e953ffe0d5b49"
server_key_iv: "7e3579abb5e3886188e004cecaa17f10"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.292621 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "3782174e37fc413ca42c3c7753b76678"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-3h01"
server_key: "0520f053c9b17948c84e953ffe0d5b49"
server_key_iv: "7e3579abb5e3886188e004cecaa17f10"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.297987 20325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:11.301522 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:11.302948 20325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:35:11.303102 20325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "3782174e37fc413ca42c3c7753b76678"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-3h01"
server_key: "0520f053c9b17948c84e953ffe0d5b49"
server_key_iv: "7e3579abb5e3886188e004cecaa17f10"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.303325 20325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:11.319178 20325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:11.323110 20325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:11.323405 20325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:11.325891 20325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:11.325996 20325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:11.326103 20325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:11.326177 20325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:35:11 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:11.348866 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.326567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:60135 (local address 127.17.205.62:34533)
0929 16:35:11.326763 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.326770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.328186 (+ 1416us) server_negotiation.cc:407] Connection header received
0929 16:35:11.330207 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.330213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.330302 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.330398 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.333017 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.333752 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.336663 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.337198 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.337331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.341421 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.341429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.341435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:11.341485 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.345055 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.346129 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.346134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.346138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.346278 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.346853 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.346858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.346861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.347280 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.347429 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.348259 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.348505 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:35:11.349932 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.327129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.3:60135)
0929 16:35:11.328015 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.328063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.329922 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.330824 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.330838 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.331754 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:11.332785 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.332818 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.333920 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.333927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.336490 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.336508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.337669 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.337678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.337977 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.338714 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:11.338737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:11.341081 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:11.345216 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.345228 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.345252 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.345945 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.346416 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.346422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.346425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.346713 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.347431 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:11.347444 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:11.347846 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.348104 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.348672 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":346,"threads_started":1}
I20260929 16:35:11.369807 20325 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43757
I20260929 16:35:11.369856 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43757 every 8 connection(s)
I20260929 16:35:11.371081 20325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:11.373157 20460 heartbeater.cc:347] Connected to a master server at 127.17.205.62:34533
I20260929 16:35:11.373389 20460 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:11.374239 20460 heartbeater.cc:511] Master 127.17.205.62:34533 requested a full tablet report, sending...
I20260929 16:35:11.375793 19975 ts_manager.cc:194] Registered new tserver with Master: 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:11.376909 19975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:60135
I20260929 16:35:11.378931 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20325
I20260929 16:35:11.379082 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:11.379544 18228 external_mini_cluster.cc:1428] Setting key 2f0ada79e39b5362e264bf15d4277163
I20260929 16:35:11.380743 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.367457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:34837 (local address 127.17.205.62:34533)
0929 16:35:11.367646 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.367653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.368184 (+ 531us) server_negotiation.cc:407] Connection header received
0929 16:35:11.368434 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.368440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.368520 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.368606 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.369623 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.370453 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.372169 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.372667 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.372800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.374782 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.374787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.374791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:11.374821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.377852 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.378702 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.378707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.378711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.378863 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.379304 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.379308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.379311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.379631 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.379754 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.380229 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.380417 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:35:11.380785 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.367158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.17.205.3:34837)
0929 16:35:11.368075 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.368095 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.368306 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.368635 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.368639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.369143 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:11.369478 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.369489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.370578 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.370583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.371985 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.371999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.372834 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.372838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.372989 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.373618 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:11.373639 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:11.374608 (+ 969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:11.377963 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.377968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.377973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.378537 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.379003 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.379008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.379011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.379150 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.379841 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:11.379844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:11.379974 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.380092 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.380386 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":633,"threads_started":1}
I20260929 16:35:11.383169 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:11.402594 20182 heartbeater.cc:503] Master 127.17.205.62:34533 was elected leader, sending a full tablet report...
Sep 29 16:35:11 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:35:11 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:11.438604 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.420826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.17.205.62:34533)
0929 16:35:11.421032 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.421038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.421437 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:35:11.421624 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.421630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.421727 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.421834 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.423787 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.424793 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.426453 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.427071 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.427173 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.431873 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.431881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.431886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:11.431950 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.435340 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.436102 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.436109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.436113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.436287 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.436720 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.436725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.436728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.437116 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.437363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.437711 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.437977 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:35:11.443791 19975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:11.451120 19975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:11.477228 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.464388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.17.205.2:42855)
0929 16:35:11.468698 (+ 4310us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.468709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.468742 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:35:11.468864 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.468873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.469182 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.469406 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.470194 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.471621 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.474592 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.475988 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.476115 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.476330 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.476753 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.476941 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4131,"thread_start_us":4014,"threads_started":1}
I20260929 16:35:11.477350 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.463680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:42855 (local address 127.0.0.1:57580)
0929 16:35:11.468036 (+ 4356us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.468053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.468154 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.469449 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.469458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.469498 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.469938 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.469958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.471776 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.471784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.474386 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.474400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.476198 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.476204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.476475 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.476541 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.476682 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.476895 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4231,"mutex_wait_us":4,"thread_start_us":3004,"threads_started":1}
I20260929 16:35:11.478820 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.463585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33137 (local address 127.0.0.1:35222)
0929 16:35:11.467138 (+ 3553us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.467193 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.467425 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.471680 (+ 4255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.471685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.471714 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.472223 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.472236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.474100 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.474109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.476270 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.476282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.478030 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.478036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.478195 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.478218 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.478316 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.478443 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3227,"mutex_wait_us":4,"thread_start_us":4149,"threads_started":1}
I20260929 16:35:11.479911 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.466402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.17.205.1:33137)
0929 16:35:11.470737 (+ 4335us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.470746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.470776 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:35:11.470894 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.470903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.471189 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.471447 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.472415 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.473943 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.476433 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.477824 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.477941 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.478135 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.478362 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.479618 (+ 1256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3421,"thread_start_us":3276,"threads_started":1}
I20260929 16:35:11.481248 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.464208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43757 (local address 127.0.0.1:41288)
0929 16:35:11.472764 (+ 8556us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.472781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.472906 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.473841 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.473845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.473868 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.474310 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.474322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.476372 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.476377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.478636 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.478649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.480546 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.480552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.480710 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.480741 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.480829 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.480955 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8431,"thread_start_us":8290,"threads_started":1}
I20260929 16:35:11.481631 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 90004f687aa1460a9564eaddf806368e (DEFAULT_TABLE table=test-table [id=31a0ecbf9db34a269ce86b840a2211fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:11.482791 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.465589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.17.205.3:43757)
0929 16:35:11.470542 (+ 4953us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.470553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.472880 (+ 2327us) server_negotiation.cc:407] Connection header received
0929 16:35:11.473044 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.473050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.473302 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.473535 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.474553 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.476194 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.478823 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.480330 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.480459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.480664 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.480886 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.482493 (+ 1607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4754,"thread_start_us":5925,"threads_started":1}
I20260929 16:35:11.482847 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90004f687aa1460a9564eaddf806368e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.483381 20112 tablet_service.cc:1467] Processing CreateTablet for tablet 90004f687aa1460a9564eaddf806368e (DEFAULT_TABLE table=test-table [id=31a0ecbf9db34a269ce86b840a2211fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:11.484777 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90004f687aa1460a9564eaddf806368e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.486392 20390 tablet_service.cc:1467] Processing CreateTablet for tablet 90004f687aa1460a9564eaddf806368e (DEFAULT_TABLE table=test-table [id=31a0ecbf9db34a269ce86b840a2211fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:11.487819 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90004f687aa1460a9564eaddf806368e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.494154 20479 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Bootstrap starting.
I20260929 16:35:11.497339 20479 tablet_bootstrap.cc:654] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.498690 20479 log.cc:824] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:11.498956 20480 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Bootstrap starting.
I20260929 16:35:11.501600 20482 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Bootstrap starting.
I20260929 16:35:11.502360 20480 tablet_bootstrap.cc:654] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.502702 20479 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: No bootstrap required, opened a new log
I20260929 16:35:11.502975 20479 ts_tablet_manager.cc:1397] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 16:35:11.503683 20480 log.cc:824] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:11.504595 20482 tablet_bootstrap.cc:654] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.505774 20482 log.cc:824] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:11.506860 20480 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: No bootstrap required, opened a new log
I20260929 16:35:11.507108 20480 ts_tablet_manager.cc:1397] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 16:35:11.508898 20482 tablet_bootstrap.cc:492] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: No bootstrap required, opened a new log
I20260929 16:35:11.509152 20482 ts_tablet_manager.cc:1397] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:35:11.511024 20479 raft_consensus.cc:348] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.511433 20479 raft_consensus.cc:374] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.511523 20479 raft_consensus.cc:729] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab27c8fa6ff4db084f48e50fb486aca, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.512094 20479 consensus_queue.cc:260] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.513293 20321 heartbeater.cc:503] Master 127.17.205.62:34533 was elected leader, sending a full tablet report...
I20260929 16:35:11.513893 20479 ts_tablet_manager.cc:1428] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:35:11.517254 20480 raft_consensus.cc:348] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.517716 20480 raft_consensus.cc:374] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.517802 20480 raft_consensus.cc:729] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.518440 20480 consensus_queue.cc:260] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.519469 20482 raft_consensus.cc:348] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.519896 20482 raft_consensus.cc:374] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.520000 20482 raft_consensus.cc:729] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.520303 20480 ts_tablet_manager.cc:1428] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260929 16:35:11.520710 20482 consensus_queue.cc:260] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.521817 20460 heartbeater.cc:503] Master 127.17.205.62:34533 was elected leader, sending a full tablet report...
I20260929 16:35:11.522326 20482 ts_tablet_manager.cc:1428] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260929 16:35:11.580916 20487 raft_consensus.cc:482] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:11.581241 20487 raft_consensus.cc:504] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.582996 20487 leader_election.cc:288] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137), 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855)
I20260929 16:35:11.588999 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.583627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:58959 (local address 127.17.205.2:42855)
0929 16:35:11.583801 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.583807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.583848 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:35:11.583937 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.583942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.584020 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.584113 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.584545 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.585460 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.587233 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.588177 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.588297 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.588350 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.588662 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.588749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:35:11.589069 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.583362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:42855 (local address 127.17.205.3:58959)
0929 16:35:11.583584 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.583605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.583736 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.584168 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.584172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.584197 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.584403 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.584414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.585595 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.585600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.587056 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.587068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.588318 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.588324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.588510 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.588547 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.588650 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.588784 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:35:11.589689 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.583480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:52199 (local address 127.17.205.1:33137)
0929 16:35:11.583663 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.583669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.584290 (+ 621us) server_negotiation.cc:407] Connection header received
0929 16:35:11.584418 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.584423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.584506 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.584581 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.585130 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.585973 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.587824 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.588780 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.588918 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.588988 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.589348 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.589476 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:35:11.589780 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.583467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33137 (local address 127.17.205.3:52199)
0929 16:35:11.584175 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.584191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.584299 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.584612 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.584616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.584646 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.584982 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.584993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.586106 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.586110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.587673 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.587686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.588991 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.588998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.589154 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.589185 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.589306 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.589437 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":1,"thread_start_us":387,"threads_started":1}
I20260929 16:35:11.590097 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90004f687aa1460a9564eaddf806368e" candidate_uuid: "3782174e37fc413ca42c3c7753b76678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" is_pre_election: true
I20260929 16:35:11.590571 20275 raft_consensus.cc:2454] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3782174e37fc413ca42c3c7753b76678 in term 0.
I20260929 16:35:11.590646 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90004f687aa1460a9564eaddf806368e" candidate_uuid: "3782174e37fc413ca42c3c7753b76678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265fcb7d1064424a8a8e484d40ccef42" is_pre_election: true
I20260929 16:35:11.591126 20132 raft_consensus.cc:2454] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3782174e37fc413ca42c3c7753b76678 in term 0.
I20260929 16:35:11.591161 20343 leader_election.cc:302] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab27c8fa6ff4db084f48e50fb486aca, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:11.591571 20487 raft_consensus.cc:2781] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:11.591715 20487 raft_consensus.cc:482] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:11.591820 20487 raft_consensus.cc:3037] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.593943 20487 raft_consensus.cc:504] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.594491 20487 leader_election.cc:288] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [CANDIDATE]: Term 1 election: Requested vote from peers 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137), 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855)
I20260929 16:35:11.594892 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90004f687aa1460a9564eaddf806368e" candidate_uuid: "3782174e37fc413ca42c3c7753b76678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265fcb7d1064424a8a8e484d40ccef42"
I20260929 16:35:11.594942 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90004f687aa1460a9564eaddf806368e" candidate_uuid: "3782174e37fc413ca42c3c7753b76678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
I20260929 16:35:11.595103 20132 raft_consensus.cc:3037] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.595111 20275 raft_consensus.cc:3037] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.598147 20275 raft_consensus.cc:2454] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3782174e37fc413ca42c3c7753b76678 in term 1.
I20260929 16:35:11.598147 20132 raft_consensus.cc:2454] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3782174e37fc413ca42c3c7753b76678 in term 1.
I20260929 16:35:11.598583 20344 leader_election.cc:302] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:11.598894 20487 raft_consensus.cc:2781] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:11.599323 20487 raft_consensus.cc:686] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 1 LEADER]: Becoming Leader. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Running, Role: LEADER
I20260929 16:35:11.599834 20487 consensus_queue.cc:237] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.604214 19975 catalog_manager.cc:5495] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3782174e37fc413ca42c3c7753b76678 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "3782174e37fc413ca42c3c7753b76678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } health_report { overall_health: HEALTHY } } }
W20260929 16:35:11.622836 20461 tablet.cc:2369] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:11.630746 20322 tablet.cc:2369] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:11.644249 20183 tablet.cc:2369] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:35:11 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699709, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:11.686048 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.665949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.17.205.62:34533)
0929 16:35:11.666146 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.666152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.670830 (+ 4678us) server_negotiation.cc:407] Connection header received
0929 16:35:11.671070 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.671076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.671141 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.671199 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.672816 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.673872 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.676165 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.676753 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.676893 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.680850 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.680856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.680860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:11.680891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.683145 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.683896 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.683902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.683906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.684048 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:11.684456 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:11.684462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:11.684466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:11.684751 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.684890 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.685363 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.685657 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":75}
I20260929 16:35:11.686082 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.666003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34533 (local address 127.0.0.1:56608)
0929 16:35:11.670710 (+ 4707us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.670733 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.670953 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.671422 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.671426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.672334 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:11.672665 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.672678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.674035 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.674040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.676014 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.676031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.676962 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.676973 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.677170 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.677916 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:11.677950 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:11.680630 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:11.683319 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.683333 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.683342 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.683755 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.684163 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:11.684169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:11.684173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:11.684341 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:11.684882 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:11.684891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:11.685079 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.685224 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.685519 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4522,"thread_start_us":7972,"threads_started":1}
I20260929 16:35:11.691329 19975 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:35:11.692309 19975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:11.698563 20112 tablet_service.cc:1467] Processing CreateTablet for tablet c54051eace504e0fbd16063c5985bee2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a588fefa21f64f91bb5c575ec837975e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:11.698599 20251 tablet_service.cc:1467] Processing CreateTablet for tablet c54051eace504e0fbd16063c5985bee2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a588fefa21f64f91bb5c575ec837975e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:11.698937 20390 tablet_service.cc:1467] Processing CreateTablet for tablet c54051eace504e0fbd16063c5985bee2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a588fefa21f64f91bb5c575ec837975e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:35:11.699616 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54051eace504e0fbd16063c5985bee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.699615 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54051eace504e0fbd16063c5985bee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.699712 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54051eace504e0fbd16063c5985bee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.705281 20482 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: Bootstrap starting.
I20260929 16:35:11.706586 20480 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: Bootstrap starting.
I20260929 16:35:11.706732 20482 tablet_bootstrap.cc:654] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.708160 20479 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: Bootstrap starting.
I20260929 16:35:11.708165 20480 tablet_bootstrap.cc:654] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.709610 20479 tablet_bootstrap.cc:654] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.710250 20480 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: No bootstrap required, opened a new log
I20260929 16:35:11.710376 20480 ts_tablet_manager.cc:1397] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 16:35:11.711095 20482 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: No bootstrap required, opened a new log
I20260929 16:35:11.711266 20482 ts_tablet_manager.cc:1397] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 16:35:11.711108 20480 raft_consensus.cc:348] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.711359 20480 raft_consensus.cc:374] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.711458 20480 raft_consensus.cc:729] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.711676 20480 consensus_queue.cc:260] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.712019 20479 tablet_bootstrap.cc:492] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: No bootstrap required, opened a new log
I20260929 16:35:11.712155 20479 ts_tablet_manager.cc:1397] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:35:11.712018 20482 raft_consensus.cc:348] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.712226 20482 raft_consensus.cc:374] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.712304 20482 raft_consensus.cc:729] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.712148 20486 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.712513 20486 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.712549 20482 consensus_queue.cc:260] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.712870 20479 raft_consensus.cc:348] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.713083 20479 raft_consensus.cc:374] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.712967 20487 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.713171 20479 raft_consensus.cc:729] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab27c8fa6ff4db084f48e50fb486aca, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.713232 20487 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.713379 20479 consensus_queue.cc:260] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.713898 20479 ts_tablet_manager.cc:1428] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:35:11.714265 20480 ts_tablet_manager.cc:1428] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:35:11.714056 20485 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.714378 20485 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.715778 20482 ts_tablet_manager.cc:1428] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:35:11.759275 20486 raft_consensus.cc:482] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:11.759523 20486 raft_consensus.cc:504] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.761080 20486 leader_election.cc:288] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:11.768563 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.761532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:38065 (local address 127.17.205.2:42855)
0929 16:35:11.761704 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.761711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.762226 (+ 515us) server_negotiation.cc:407] Connection header received
0929 16:35:11.762423 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.762427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.762488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.762562 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.763127 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.763924 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.766431 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.767627 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.767811 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.767899 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.768103 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.768240 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:35:11.768590 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.761438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:42855 (local address 127.17.205.1:38065)
0929 16:35:11.762106 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.762125 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.762282 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.762598 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.762603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.762634 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.762982 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.762991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.764085 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.764089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.766261 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.766276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.767826 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.767831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.767955 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.767979 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.768050 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.768160 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"spinlock_wait_cycles":32512,"thread_start_us":402,"threads_started":1}
I20260929 16:35:11.768977 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.761532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:58819 (local address 127.17.205.3:43757)
0929 16:35:11.761713 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.761718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.762751 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 16:35:11.762959 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.762963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.763026 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.763107 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:11.763719 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.764512 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.766799 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.768052 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.768223 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.768299 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.768613 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.768747 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:35:11.768977 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.761512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43757 (local address 127.17.205.1:58819)
0929 16:35:11.762622 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.762643 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.762809 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.763138 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.763143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.763168 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:11.763543 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.763552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.764694 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.764700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.766631 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.766641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.768261 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.768268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.768448 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.768478 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.768573 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.768695 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":710,"threads_started":1}
I20260929 16:35:11.769574 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54051eace504e0fbd16063c5985bee2" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" is_pre_election: true
I20260929 16:35:11.769833 20275 raft_consensus.cc:2454] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:11.769826 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54051eace504e0fbd16063c5985bee2" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678" is_pre_election: true
I20260929 16:35:11.770207 20411 raft_consensus.cc:2454] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:11.770311 20064 leader_election.cc:302] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca; no voters:
I20260929 16:35:11.770787 20486 raft_consensus.cc:2781] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:11.770890 20486 raft_consensus.cc:482] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:11.770988 20486 raft_consensus.cc:3037] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.773030 20486 raft_consensus.cc:504] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.773576 20486 leader_election.cc:288] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:11.774113 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54051eace504e0fbd16063c5985bee2" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678"
I20260929 16:35:11.774093 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54051eace504e0fbd16063c5985bee2" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
I20260929 16:35:11.774283 20411 raft_consensus.cc:3037] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.774319 20275 raft_consensus.cc:3037] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.776425 20411 raft_consensus.cc:2454] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:11.776423 20275 raft_consensus.cc:2454] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:11.776834 20063 leader_election.cc:302] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:11.777166 20486 raft_consensus.cc:2781] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:11.777607 20486 raft_consensus.cc:686] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Becoming Leader. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Running, Role: LEADER
I20260929 16:35:11.778100 20486 consensus_queue.cc:237] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:11.781848 20502 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: TxnStatusTablet state changed. Reason: New leader 265fcb7d1064424a8a8e484d40ccef42. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.782143 20502 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:11.782838 20504 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:35:11.782702 19975 catalog_manager.cc:5495] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } health_report { overall_health: UNKNOWN } } }
I20260929 16:35:11.783119 20504 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:35:11.814777 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.809030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.17.205.1:33137)
0929 16:35:11.809201 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.809207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.809483 (+ 276us) server_negotiation.cc:407] Connection header received
0929 16:35:11.809742 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.809746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.809862 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.809993 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:11.810355 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.811139 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.812439 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.812904 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.813012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.813508 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.814130 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.814335 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.814381 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.814464 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:11.814821 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.809041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33137 (local address 127.0.0.1:35224)
0929 16:35:11.809385 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.809401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.809546 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.810029 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.810033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.810045 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:11.810230 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.810238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.811278 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.811284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.812308 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.812320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.813070 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.813077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.813272 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.813328 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.814228 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.814258 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.814327 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.814484 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260929 16:35:11.822408 20411 raft_consensus.cc:1261] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:11.822643 20275 raft_consensus.cc:1261] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:11.823246 20486 consensus_queue.cc:1035] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:11.823539 20502 consensus_queue.cc:1035] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:11.837543 20487 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: TxnStatusTablet state changed. Reason: New leader 265fcb7d1064424a8a8e484d40ccef42. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.837878 20487 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.838110 20485 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: TxnStatusTablet state changed. Reason: New leader 265fcb7d1064424a8a8e484d40ccef42. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.838402 20485 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.852434 20486 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.852772 20486 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:11.853902 20502 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.854172 20502 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 265fcb7d1064424a8a8e484d40ccef42: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:35:11.855193 20487 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.855481 20487 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 3782174e37fc413ca42c3c7753b76678: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.860989 20485 tablet_replica.cc:439] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } } }
I20260929 16:35:11.861243 20485 tablet_replica.cc:442] T c54051eace504e0fbd16063c5985bee2 P 2ab27c8fa6ff4db084f48e50fb486aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:35:11.869427 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.863537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.17.205.3:43757)
0929 16:35:11.863716 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.863722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.863858 (+ 136us) server_negotiation.cc:407] Connection header received
0929 16:35:11.864071 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.864075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.864155 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.864269 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:11.864649 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.865451 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.867349 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.867730 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.867818 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.868317 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.868868 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.868982 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.869088 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.869157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:35:11.886193 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.880519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33137 (local address 127.17.205.3:41503)
0929 16:35:11.880756 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.880776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.880921 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.881310 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.881314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.881326 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:11.881505 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.881514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.883171 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.883177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.884235 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.884248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.885084 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.885090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.885219 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.885274 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.885669 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.885680 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.885753 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.885908 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:35:11.886179 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.880585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:41503 (local address 127.17.205.1:33137)
0929 16:35:11.880756 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.880762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.880870 (+ 108us) server_negotiation.cc:407] Connection header received
0929 16:35:11.881084 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.881088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.881180 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.881269 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:11.882228 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.883033 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.884380 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.884939 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.885049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.885479 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.885584 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.885654 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.885788 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.885877 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:11.898481 20275 raft_consensus.cc:1261] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Refusing update from remote peer 3782174e37fc413ca42c3c7753b76678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:11.898531 20132 raft_consensus.cc:1261] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Refusing update from remote peer 3782174e37fc413ca42c3c7753b76678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:11.899348 20487 consensus_queue.cc:1035] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:11.900015 20487 consensus_queue.cc:1035] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:11.946538 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.940450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:37667 (local address 127.17.205.3:43757)
0929 16:35:11.940635 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.940641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.941272 (+ 631us) server_negotiation.cc:407] Connection header received
0929 16:35:11.941478 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.941484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.941572 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.941662 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:11.942282 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.943078 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.944801 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.945222 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.945332 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.945756 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.945902 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.945998 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.946179 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.946264 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:11.946934 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.940459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43757 (local address 127.17.205.1:37667)
0929 16:35:11.941168 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:11.941187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:11.941346 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:11.941679 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:11.941684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:11.941698 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:11.942096 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.942108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.943229 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.943235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.944667 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:11.944682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:11.945397 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:11.945407 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:11.945529 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:11.945592 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:11.946024 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:11.946037 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 16:35:11.946127 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:35:11.946283 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":359,"threads_started":1}
I20260929 16:35:11.982259 19975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:11.982550 19975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:11.986497 19975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Sending DeleteTablet for 3 replicas of tablet 90004f687aa1460a9564eaddf806368e
I20260929 16:35:11.987694 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet 90004f687aa1460a9564eaddf806368e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:35222
I20260929 16:35:11.987692 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet 90004f687aa1460a9564eaddf806368e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:57580
I20260929 16:35:11.988910 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet 90004f687aa1460a9564eaddf806368e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:41288
I20260929 16:35:11.995693 19975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:11.996388 19975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:11.997030 20527 tablet_replica.cc:330] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: stopping tablet replica
I20260929 16:35:11.997396 20528 tablet_replica.cc:330] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: stopping tablet replica
I20260929 16:35:12.004750 20389 tablet_service.cc:1467] Processing CreateTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 (DEFAULT_TABLE table=test-table [id=d04186229e1f4efa96f9271efc421883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:12.005359 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760cdd6e2ff54c66bce073f5463f0724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:12.004845 20112 tablet_service.cc:1467] Processing CreateTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 (DEFAULT_TABLE table=test-table [id=d04186229e1f4efa96f9271efc421883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:12.005867 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760cdd6e2ff54c66bce073f5463f0724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:12.008914 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 (DEFAULT_TABLE table=test-table [id=d04186229e1f4efa96f9271efc421883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:12.009384 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760cdd6e2ff54c66bce073f5463f0724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:12.009779 20527 raft_consensus.cc:2229] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:12.010103 20528 raft_consensus.cc:2229] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:12.010558 20527 raft_consensus.cc:2258] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:12.012882 20528 raft_consensus.cc:2258] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:12.016109 20528 ts_tablet_manager.cc:1905] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:12.017158 20527 ts_tablet_manager.cc:1905] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:12.023334 20482 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Bootstrap starting.
I20260929 16:35:12.025159 20482 tablet_bootstrap.cc:654] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:12.019186 20480 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Bootstrap starting.
I20260929 16:35:12.027940 20480 tablet_bootstrap.cc:654] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:12.031765 20527 ts_tablet_manager.cc:1918] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:12.033598 20527 log.cc:1196] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/90004f687aa1460a9564eaddf806368e
I20260929 16:35:12.034215 20527 ts_tablet_manager.cc:1939] T 90004f687aa1460a9564eaddf806368e P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting consensus metadata
I20260929 16:35:12.036212 19961 catalog_manager.cc:4841] TS 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855): tablet 90004f687aa1460a9564eaddf806368e (table test-table [id=31a0ecbf9db34a269ce86b840a2211fa]) successfully deleted
I20260929 16:35:12.037295 20528 ts_tablet_manager.cc:1918] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:12.037488 20528 log.cc:1196] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/90004f687aa1460a9564eaddf806368e
I20260929 16:35:12.038050 20528 ts_tablet_manager.cc:1939] T 90004f687aa1460a9564eaddf806368e P 265fcb7d1064424a8a8e484d40ccef42: Deleting consensus metadata
I20260929 16:35:12.039819 19962 catalog_manager.cc:4841] TS 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137): tablet 90004f687aa1460a9564eaddf806368e (table test-table [id=31a0ecbf9db34a269ce86b840a2211fa]) successfully deleted
I20260929 16:35:12.041070 20479 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Bootstrap starting.
I20260929 16:35:12.042709 20479 tablet_bootstrap.cc:654] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:12.053960 20482 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: No bootstrap required, opened a new log
I20260929 16:35:12.054171 20482 ts_tablet_manager.cc:1397] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.006s
I20260929 16:35:12.055160 20479 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: No bootstrap required, opened a new log
I20260929 16:35:12.055361 20479 ts_tablet_manager.cc:1397] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260929 16:35:12.055665 20529 tablet_replica.cc:330] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: stopping tablet replica
I20260929 16:35:12.056136 20529 raft_consensus.cc:2229] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:12.056102 20479 raft_consensus.cc:348] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.056342 20479 raft_consensus.cc:374] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:12.056423 20479 raft_consensus.cc:729] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab27c8fa6ff4db084f48e50fb486aca, State: Initialized, Role: FOLLOWER
I20260929 16:35:12.056797 20529 raft_consensus.cc:2258] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:12.056713 20479 consensus_queue.cc:260] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.055047 20482 raft_consensus.cc:348] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.057284 20482 raft_consensus.cc:374] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:12.057384 20482 raft_consensus.cc:729] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Initialized, Role: FOLLOWER
I20260929 16:35:12.057626 20482 consensus_queue.cc:260] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.059113 20482 ts_tablet_manager.cc:1428] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:35:12.060707 20479 ts_tablet_manager.cc:1428] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:35:12.062779 20529 ts_tablet_manager.cc:1905] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:12.063948 20480 tablet_bootstrap.cc:492] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: No bootstrap required, opened a new log
I20260929 16:35:12.064128 20480 ts_tablet_manager.cc:1397] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.007s
I20260929 16:35:12.064829 20480 raft_consensus.cc:348] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.065064 20480 raft_consensus.cc:374] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:12.065135 20480 raft_consensus.cc:729] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Initialized, Role: FOLLOWER
I20260929 16:35:12.065380 20480 consensus_queue.cc:260] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.065658 20512 raft_consensus.cc:482] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:12.065979 20480 ts_tablet_manager.cc:1428] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:35:12.065868 20512 raft_consensus.cc:504] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.066645 20512 leader_election.cc:288] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:12.067126 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760cdd6e2ff54c66bce073f5463f0724" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678" is_pre_election: true
I20260929 16:35:12.067382 20411 raft_consensus.cc:2454] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:12.067329 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760cdd6e2ff54c66bce073f5463f0724" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" is_pre_election: true
I20260929 16:35:12.067523 20275 raft_consensus.cc:2454] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:12.067930 20063 leader_election.cc:302] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:12.068411 20512 raft_consensus.cc:2781] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:12.068529 20512 raft_consensus.cc:482] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:12.068612 20512 raft_consensus.cc:3037] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:12.071633 20512 raft_consensus.cc:504] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.072652 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760cdd6e2ff54c66bce073f5463f0724" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
I20260929 16:35:12.072849 20275 raft_consensus.cc:3037] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:12.073364 20529 ts_tablet_manager.cc:1918] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:12.073536 20529 log.cc:1196] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/90004f687aa1460a9564eaddf806368e
I20260929 16:35:12.074030 20529 ts_tablet_manager.cc:1939] T 90004f687aa1460a9564eaddf806368e P 3782174e37fc413ca42c3c7753b76678: Deleting consensus metadata
I20260929 16:35:12.075685 20275 raft_consensus.cc:2454] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:12.075736 19960 catalog_manager.cc:4841] TS 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757): tablet 90004f687aa1460a9564eaddf806368e (table test-table [id=31a0ecbf9db34a269ce86b840a2211fa]) successfully deleted
I20260929 16:35:12.076154 20064 leader_election.cc:302] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca; no voters:
I20260929 16:35:12.076453 20502 raft_consensus.cc:2781] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:12.076601 20502 raft_consensus.cc:686] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Becoming Leader. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Running, Role: LEADER
I20260929 16:35:12.076992 20512 leader_election.cc:288] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:12.076857 20502 consensus_queue.cc:237] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:12.078270 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760cdd6e2ff54c66bce073f5463f0724" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678"
I20260929 16:35:12.078485 20411 raft_consensus.cc:3037] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:12.081555 20411 raft_consensus.cc:2454] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:12.081557 19975 catalog_manager.cc:5495] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } health_report { overall_health: UNKNOWN } } }
I20260929 16:35:12.110625 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:12.103907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.17.205.1:33137)
0929 16:35:12.104221 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:12.104227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:12.104481 (+ 254us) server_negotiation.cc:407] Connection header received
0929 16:35:12.104687 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:12.104692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:12.104780 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:12.104884 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:12.105253 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.106416 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.108480 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.109198 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.109334 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:12.109837 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:12.109991 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:12.110081 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:12.110214 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:35:12.110299 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260929 16:35:12.127381 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:12.121304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:34821 (local address 127.17.205.1:33137)
0929 16:35:12.121461 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:12.121467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:12.121496 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:35:12.121553 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:12.121558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:12.121642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:12.121733 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:12.122188 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.123386 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.125304 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.125956 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.126072 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:12.126500 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:12.126640 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:12.126722 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:12.126880 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:35:12.126999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":48}
I20260929 16:35:12.127406 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:12.121014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33137 (local address 127.17.205.1:34821)
0929 16:35:12.121245 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:12.121266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:12.121387 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:12.121778 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:12.121783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:12.121796 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:35:12.122041 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:12.122054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.123537 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.123543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:12.125162 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:12.125176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:12.126108 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:12.126114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:12.126280 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:12.126328 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:35:12.126741 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:35:12.126752 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:35:12.126851 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:35:12.127050 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":63}
I20260929 16:35:12.137090 20275 raft_consensus.cc:1261] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:12.137162 20410 raft_consensus.cc:1261] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:12.137813 20502 consensus_queue.cc:1035] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:12.138105 20519 consensus_queue.cc:1035] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:12.155705 20507 mvcc.cc:205] Tried to move back new op lower bound from 7334706020907986944 to 7334706020669341696. Current Snapshot: MvccSnapshot[applied={T|T < 7334706020907986944}]
I20260929 16:35:12.159065 20510 mvcc.cc:205] Tried to move back new op lower bound from 7334706020907986944 to 7334706020669341696. Current Snapshot: MvccSnapshot[applied={T|T < 7334706020907986944}]
I20260929 16:35:12.181231 20508 mvcc.cc:205] Tried to move back new op lower bound from 7334706020907986944 to 7334706020669341696. Current Snapshot: MvccSnapshot[applied={T|T < 7334706020907986944}]
W20260929 16:35:12.190094 20507 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:35:13.786845 19973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:13.787081 19973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:13.791347 19973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Sending DeleteTablet for 3 replicas of tablet 760cdd6e2ff54c66bce073f5463f0724
I20260929 16:35:13.792161 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:57580
I20260929 16:35:13.792161 20111 tablet_service.cc:1514] Processing DeleteTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:35222
I20260929 16:35:13.792240 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet 760cdd6e2ff54c66bce073f5463f0724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:41288
I20260929 16:35:13.805001 20559 tablet_replica.cc:330] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: stopping tablet replica
I20260929 16:35:13.805439 20559 raft_consensus.cc:2229] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:13.805799 20559 raft_consensus.cc:2258] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:13.806457 20558 tablet_replica.cc:330] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: stopping tablet replica
I20260929 16:35:13.807364 20559 ts_tablet_manager.cc:1905] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:13.807775 20560 tablet_replica.cc:330] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: stopping tablet replica
I20260929 16:35:13.807971 20560 raft_consensus.cc:2229] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:13.808209 20560 raft_consensus.cc:2258] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:13.809198 20560 ts_tablet_manager.cc:1905] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:13.809486 20558 raft_consensus.cc:2229] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:13.811784 20558 raft_consensus.cc:2258] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:13.813695 20558 ts_tablet_manager.cc:1905] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:13.815667 20560 ts_tablet_manager.cc:1918] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:13.815795 20560 log.cc:1196] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/760cdd6e2ff54c66bce073f5463f0724
I20260929 16:35:13.816170 20560 ts_tablet_manager.cc:1939] T 760cdd6e2ff54c66bce073f5463f0724 P 3782174e37fc413ca42c3c7753b76678: Deleting consensus metadata
I20260929 16:35:13.817268 19960 catalog_manager.cc:4841] TS 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757): tablet 760cdd6e2ff54c66bce073f5463f0724 (table test-table [id=d04186229e1f4efa96f9271efc421883]) successfully deleted
I20260929 16:35:13.819283 20559 ts_tablet_manager.cc:1918] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:13.819463 20559 log.cc:1196] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/760cdd6e2ff54c66bce073f5463f0724
I20260929 16:35:13.819931 20559 ts_tablet_manager.cc:1939] T 760cdd6e2ff54c66bce073f5463f0724 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting consensus metadata
I20260929 16:35:13.821074 19961 catalog_manager.cc:4841] TS 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855): tablet 760cdd6e2ff54c66bce073f5463f0724 (table test-table [id=d04186229e1f4efa96f9271efc421883]) successfully deleted
I20260929 16:35:13.822304 20558 ts_tablet_manager.cc:1918] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:13.822431 20558 log.cc:1196] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/760cdd6e2ff54c66bce073f5463f0724
I20260929 16:35:13.822824 20558 ts_tablet_manager.cc:1939] T 760cdd6e2ff54c66bce073f5463f0724 P 265fcb7d1064424a8a8e484d40ccef42: Deleting consensus metadata
I20260929 16:35:13.823908 19962 catalog_manager.cc:4841] TS 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137): tablet 760cdd6e2ff54c66bce073f5463f0724 (table test-table [id=d04186229e1f4efa96f9271efc421883]) successfully deleted
Sep 29 16:35:13 dist-test-slave-3h01 krb5kdc[19930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:35:13 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:13.861613 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:13.840555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.17.205.62:34533)
0929 16:35:13.841298 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:13.841304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:13.841328 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:35:13.841391 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:13.841395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:13.841457 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:13.841550 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:13.842850 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.843698 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.844963 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.845352 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.845467 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:13.856169 (+ 10702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:13.856175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:13.856179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:13.856221 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:13.858899 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.859665 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.859670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.859673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.859836 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.860227 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.860231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.860233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.860530 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:13.860761 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:13.861043 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:35:13.861252 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":454,"threads_started":1}
I20260929 16:35:13.865295 19973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:13.865934 19973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:13.872234 20111 tablet_service.cc:1467] Processing CreateTablet for tablet 69c9b2c9af404140af222ada7c77a638 (DEFAULT_TABLE table=test-table [id=5c632089a20442acb8bfee3550aa46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:13.872299 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 69c9b2c9af404140af222ada7c77a638 (DEFAULT_TABLE table=test-table [id=5c632089a20442acb8bfee3550aa46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:13.872593 20390 tablet_service.cc:1467] Processing CreateTablet for tablet 69c9b2c9af404140af222ada7c77a638 (DEFAULT_TABLE table=test-table [id=5c632089a20442acb8bfee3550aa46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:13.872725 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c9b2c9af404140af222ada7c77a638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:13.872750 20111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c9b2c9af404140af222ada7c77a638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:13.872954 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c9b2c9af404140af222ada7c77a638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:13.885720 20571 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Bootstrap starting.
I20260929 16:35:13.886274 20569 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Bootstrap starting.
I20260929 16:35:13.887591 20571 tablet_bootstrap.cc:654] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:13.887925 20569 tablet_bootstrap.cc:654] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:13.890702 20570 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Bootstrap starting.
I20260929 16:35:13.892664 20570 tablet_bootstrap.cc:654] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:13.904198 20571 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: No bootstrap required, opened a new log
I20260929 16:35:13.904291 20569 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: No bootstrap required, opened a new log
I20260929 16:35:13.904364 20571 ts_tablet_manager.cc:1397] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260929 16:35:13.904445 20569 ts_tablet_manager.cc:1397] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.006s
I20260929 16:35:13.904876 20570 tablet_bootstrap.cc:492] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: No bootstrap required, opened a new log
I20260929 16:35:13.905095 20570 ts_tablet_manager.cc:1397] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.007s
I20260929 16:35:13.905162 20571 raft_consensus.cc:348] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.905229 20569 raft_consensus.cc:348] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.905414 20571 raft_consensus.cc:374] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:13.905470 20569 raft_consensus.cc:374] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:13.905493 20571 raft_consensus.cc:729] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab27c8fa6ff4db084f48e50fb486aca, State: Initialized, Role: FOLLOWER
I20260929 16:35:13.905565 20569 raft_consensus.cc:729] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Initialized, Role: FOLLOWER
I20260929 16:35:13.905794 20571 consensus_queue.cc:260] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.905792 20569 consensus_queue.cc:260] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.906232 20570 raft_consensus.cc:348] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.906530 20570 raft_consensus.cc:374] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:13.906661 20570 raft_consensus.cc:729] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Initialized, Role: FOLLOWER
I20260929 16:35:13.907021 20569 ts_tablet_manager.cc:1428] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 16:35:13.906932 20570 consensus_queue.cc:260] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:13.907661 20571 ts_tablet_manager.cc:1428] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 16:35:13.908401 20570 ts_tablet_manager.cc:1428] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 16:35:14.109614 20541 raft_consensus.cc:482] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:14.109833 20541 raft_consensus.cc:504] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.110526 20541 leader_election.cc:288] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:14.111052 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c9b2c9af404140af222ada7c77a638" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678" is_pre_election: true
I20260929 16:35:14.111104 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c9b2c9af404140af222ada7c77a638" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" is_pre_election: true
I20260929 16:35:14.111342 20275 raft_consensus.cc:2454] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:14.111357 20411 raft_consensus.cc:2454] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:14.111837 20064 leader_election.cc:302] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:14.112112 20541 raft_consensus.cc:2781] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:14.112213 20541 raft_consensus.cc:482] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:14.112296 20541 raft_consensus.cc:3037] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.114336 20541 raft_consensus.cc:504] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.114833 20541 leader_election.cc:288] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:14.115345 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c9b2c9af404140af222ada7c77a638" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
I20260929 16:35:14.115339 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c9b2c9af404140af222ada7c77a638" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678"
I20260929 16:35:14.115556 20275 raft_consensus.cc:3037] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.115574 20411 raft_consensus.cc:3037] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.118387 20275 raft_consensus.cc:2454] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:14.118374 20411 raft_consensus.cc:2454] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:14.118822 20064 leader_election.cc:302] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:14.119113 20541 raft_consensus.cc:2781] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:14.119267 20541 raft_consensus.cc:686] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Becoming Leader. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Running, Role: LEADER
I20260929 16:35:14.119530 20541 consensus_queue.cc:237] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.122037 19975 catalog_manager.cc:5495] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } health_report { overall_health: UNKNOWN } } }
I20260929 16:35:14.196182 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:14.190871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.17.205.1:33137)
0929 16:35:14.191468 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:14.191475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:14.191498 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:35:14.191554 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:14.191557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:14.191621 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:14.191711 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:14.192118 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.192928 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.194341 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.194778 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.194907 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:14.195416 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:14.195525 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:14.195607 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:14.195776 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:35:14.195868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
I20260929 16:35:14.204881 20275 raft_consensus.cc:1261] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:14.204969 20411 raft_consensus.cc:1261] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:14.205394 20541 consensus_queue.cc:1035] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:14.205668 20553 consensus_queue.cc:1035] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:14.245293 19975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:14.245509 19975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:14.248479 19975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Sending DeleteTablet for 3 replicas of tablet 69c9b2c9af404140af222ada7c77a638
I20260929 16:35:14.249186 20111 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c9b2c9af404140af222ada7c77a638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:14 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:35222
I20260929 16:35:14.249287 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c9b2c9af404140af222ada7c77a638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:14 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:57580
I20260929 16:35:14.249332 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c9b2c9af404140af222ada7c77a638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:14 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:41288
I20260929 16:35:14.249490 20558 tablet_replica.cc:330] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: stopping tablet replica
I20260929 16:35:14.249614 20560 tablet_replica.cc:330] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: stopping tablet replica
I20260929 16:35:14.249689 20558 raft_consensus.cc:2229] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:14.249792 20560 raft_consensus.cc:2229] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:14.249979 20560 raft_consensus.cc:2258] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:14.249995 20558 raft_consensus.cc:2258] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:14.250247 20559 tablet_replica.cc:330] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: stopping tablet replica
I20260929 16:35:14.250474 20559 raft_consensus.cc:2229] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:14.250664 20559 raft_consensus.cc:2258] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:14.250486 19975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:14.251067 19975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:14.251104 20560 ts_tablet_manager.cc:1905] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:14.251672 20559 ts_tablet_manager.cc:1905] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:14.251809 20558 ts_tablet_manager.cc:1905] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:14.256397 20558 ts_tablet_manager.cc:1918] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:14.256556 20558 log.cc:1196] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/69c9b2c9af404140af222ada7c77a638
I20260929 16:35:14.256949 20558 ts_tablet_manager.cc:1939] T 69c9b2c9af404140af222ada7c77a638 P 265fcb7d1064424a8a8e484d40ccef42: Deleting consensus metadata
I20260929 16:35:14.257778 20559 ts_tablet_manager.cc:1918] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:14.257923 20559 log.cc:1196] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/69c9b2c9af404140af222ada7c77a638
I20260929 16:35:14.258020 19962 catalog_manager.cc:4841] TS 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137): tablet 69c9b2c9af404140af222ada7c77a638 (table test-table [id=5c632089a20442acb8bfee3550aa46cd]) successfully deleted
I20260929 16:35:14.258360 20559 ts_tablet_manager.cc:1939] T 69c9b2c9af404140af222ada7c77a638 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting consensus metadata
I20260929 16:35:14.258728 20560 ts_tablet_manager.cc:1918] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:35:14.258893 20560 log.cc:1196] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/69c9b2c9af404140af222ada7c77a638
I20260929 16:35:14.259276 20560 ts_tablet_manager.cc:1939] T 69c9b2c9af404140af222ada7c77a638 P 3782174e37fc413ca42c3c7753b76678: Deleting consensus metadata
I20260929 16:35:14.259323 19961 catalog_manager.cc:4841] TS 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855): tablet 69c9b2c9af404140af222ada7c77a638 (table test-table [id=5c632089a20442acb8bfee3550aa46cd]) successfully deleted
I20260929 16:35:14.260078 19960 catalog_manager.cc:4841] TS 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757): tablet 69c9b2c9af404140af222ada7c77a638 (table test-table [id=5c632089a20442acb8bfee3550aa46cd]) successfully deleted
I20260929 16:35:14.261286 20111 tablet_service.cc:1467] Processing CreateTablet for tablet d6e19461fb544178b094ad4a043c7a98 (DEFAULT_TABLE table=test-table [id=bca0231a8b5f4f70861320493dd8d28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:14.261512 20251 tablet_service.cc:1467] Processing CreateTablet for tablet d6e19461fb544178b094ad4a043c7a98 (DEFAULT_TABLE table=test-table [id=bca0231a8b5f4f70861320493dd8d28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:14.261624 20111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e19461fb544178b094ad4a043c7a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:14.261845 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e19461fb544178b094ad4a043c7a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:14.266369 20571 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Bootstrap starting.
I20260929 16:35:14.266541 20569 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Bootstrap starting.
I20260929 16:35:14.267304 20390 tablet_service.cc:1467] Processing CreateTablet for tablet d6e19461fb544178b094ad4a043c7a98 (DEFAULT_TABLE table=test-table [id=bca0231a8b5f4f70861320493dd8d28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:14.267701 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e19461fb544178b094ad4a043c7a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:14.267817 20569 tablet_bootstrap.cc:654] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:14.267876 20571 tablet_bootstrap.cc:654] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:14.269387 20569 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: No bootstrap required, opened a new log
I20260929 16:35:14.269531 20569 ts_tablet_manager.cc:1397] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:14.270093 20569 raft_consensus.cc:348] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.270282 20569 raft_consensus.cc:374] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:14.270355 20569 raft_consensus.cc:729] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Initialized, Role: FOLLOWER
I20260929 16:35:14.270586 20569 consensus_queue.cc:260] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.270987 20569 ts_tablet_manager.cc:1428] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:35:14.272384 20571 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: No bootstrap required, opened a new log
I20260929 16:35:14.272552 20571 ts_tablet_manager.cc:1397] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260929 16:35:14.273224 20571 raft_consensus.cc:348] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.273458 20571 raft_consensus.cc:374] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:14.273535 20571 raft_consensus.cc:729] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab27c8fa6ff4db084f48e50fb486aca, State: Initialized, Role: FOLLOWER
I20260929 16:35:14.273777 20571 consensus_queue.cc:260] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.273967 20570 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Bootstrap starting.
I20260929 16:35:14.274969 20571 ts_tablet_manager.cc:1428] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:35:14.275504 20570 tablet_bootstrap.cc:654] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:14.277402 20570 tablet_bootstrap.cc:492] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: No bootstrap required, opened a new log
I20260929 16:35:14.277565 20570 ts_tablet_manager.cc:1397] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:14.278204 20570 raft_consensus.cc:348] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.278385 20570 raft_consensus.cc:374] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:14.278461 20570 raft_consensus.cc:729] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782174e37fc413ca42c3c7753b76678, State: Initialized, Role: FOLLOWER
I20260929 16:35:14.278669 20570 consensus_queue.cc:260] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.279119 20570 ts_tablet_manager.cc:1428] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:35:14.370014 20553 raft_consensus.cc:482] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:14.370277 20553 raft_consensus.cc:504] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.370978 20553 leader_election.cc:288] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:14.371536 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e19461fb544178b094ad4a043c7a98" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" is_pre_election: true
I20260929 16:35:14.371778 20275 raft_consensus.cc:2454] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:14.371680 20409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e19461fb544178b094ad4a043c7a98" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678" is_pre_election: true
I20260929 16:35:14.371955 20409 raft_consensus.cc:2454] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 0.
I20260929 16:35:14.372179 20064 leader_election.cc:302] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 2ab27c8fa6ff4db084f48e50fb486aca; no voters:
I20260929 16:35:14.372547 20553 raft_consensus.cc:2781] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:14.372637 20553 raft_consensus.cc:482] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:14.372740 20553 raft_consensus.cc:3037] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.374688 20553 raft_consensus.cc:504] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.375330 20553 leader_election.cc:288] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855), 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757)
I20260929 16:35:14.375708 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e19461fb544178b094ad4a043c7a98" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab27c8fa6ff4db084f48e50fb486aca"
I20260929 16:35:14.375819 20409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e19461fb544178b094ad4a043c7a98" candidate_uuid: "265fcb7d1064424a8a8e484d40ccef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3782174e37fc413ca42c3c7753b76678"
I20260929 16:35:14.375916 20275 raft_consensus.cc:3037] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.376019 20409 raft_consensus.cc:3037] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.378167 20409 raft_consensus.cc:2454] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:14.378629 20063 leader_election.cc:302] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265fcb7d1064424a8a8e484d40ccef42, 3782174e37fc413ca42c3c7753b76678; no voters:
I20260929 16:35:14.378803 20275 raft_consensus.cc:2454] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265fcb7d1064424a8a8e484d40ccef42 in term 1.
I20260929 16:35:14.378906 20553 raft_consensus.cc:2781] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:14.379081 20553 raft_consensus.cc:686] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Becoming Leader. State: Replica: 265fcb7d1064424a8a8e484d40ccef42, State: Running, Role: LEADER
I20260929 16:35:14.379372 20553 consensus_queue.cc:237] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } }
I20260929 16:35:14.382202 19975 catalog_manager.cc:5495] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "265fcb7d1064424a8a8e484d40ccef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265fcb7d1064424a8a8e484d40ccef42" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 } health_report { overall_health: UNKNOWN } } }
I20260929 16:35:14.415699 20269 raft_consensus.cc:1261] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:14.416189 20409 raft_consensus.cc:1261] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Refusing update from remote peer 265fcb7d1064424a8a8e484d40ccef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:14.416204 20553 consensus_queue.cc:1035] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab27c8fa6ff4db084f48e50fb486aca" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:14.416682 20553 consensus_queue.cc:1035] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3782174e37fc413ca42c3c7753b76678" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:35:14.435451 20539 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:35:16.011490 19973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:16.011765 19973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:16.015904 19973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49336ef52e34b09ba1a350a12101527: Sending DeleteTablet for 3 replicas of tablet d6e19461fb544178b094ad4a043c7a98
I20260929 16:35:16.016824 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e19461fb544178b094ad4a043c7a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:16 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:57580
I20260929 16:35:16.016870 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e19461fb544178b094ad4a043c7a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:16 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:35222
I20260929 16:35:16.016898 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e19461fb544178b094ad4a043c7a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:16 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:41288
I20260929 16:35:16.018826 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20046
I20260929 16:35:16.029330 20609 tablet_replica.cc:330] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: stopping tablet replica
I20260929 16:35:16.029572 20609 raft_consensus.cc:2229] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:16.029812 20609 raft_consensus.cc:2258] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:16.030332 20611 tablet_replica.cc:330] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: stopping tablet replica
I20260929 16:35:16.030530 20611 raft_consensus.cc:2229] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:16.030905 20611 raft_consensus.cc:2258] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:16.031188 20609 ts_tablet_manager.cc:1905] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:16.031497 20610 tablet_replica.cc:330] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: stopping tablet replica
I20260929 16:35:16.031695 20610 raft_consensus.cc:2229] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:16.031881 20611 ts_tablet_manager.cc:1905] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:16.031925 20610 raft_consensus.cc:2258] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:16.033782 20610 ts_tablet_manager.cc:1905] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:16.037509 20609 ts_tablet_manager.cc:1918] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:16.037648 20609 log.cc:1196] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/d6e19461fb544178b094ad4a043c7a98
I20260929 16:35:16.038123 20609 ts_tablet_manager.cc:1939] T d6e19461fb544178b094ad4a043c7a98 P 2ab27c8fa6ff4db084f48e50fb486aca: Deleting consensus metadata
I20260929 16:35:16.038216 20611 ts_tablet_manager.cc:1918] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:16.038338 20611 log.cc:1196] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/d6e19461fb544178b094ad4a043c7a98
I20260929 16:35:16.038794 20611 ts_tablet_manager.cc:1939] T d6e19461fb544178b094ad4a043c7a98 P 265fcb7d1064424a8a8e484d40ccef42: Deleting consensus metadata
I20260929 16:35:16.039338 19961 catalog_manager.cc:4841] TS 2ab27c8fa6ff4db084f48e50fb486aca (127.17.205.2:42855): tablet d6e19461fb544178b094ad4a043c7a98 (table test-table [id=bca0231a8b5f4f70861320493dd8d28e]) successfully deleted
I20260929 16:35:16.039983 19962 catalog_manager.cc:4841] TS 265fcb7d1064424a8a8e484d40ccef42 (127.17.205.1:33137): tablet d6e19461fb544178b094ad4a043c7a98 (table test-table [id=bca0231a8b5f4f70861320493dd8d28e]) successfully deleted
I20260929 16:35:16.040475 20610 ts_tablet_manager.cc:1918] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:35:16.040593 20610 log.cc:1196] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/d6e19461fb544178b094ad4a043c7a98
I20260929 16:35:16.040938 20610 ts_tablet_manager.cc:1939] T d6e19461fb544178b094ad4a043c7a98 P 3782174e37fc413ca42c3c7753b76678: Deleting consensus metadata
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:16.041854 19960 catalog_manager.cc:4841] TS 3782174e37fc413ca42c3c7753b76678 (127.17.205.3:43757): tablet d6e19461fb544178b094ad4a043c7a98 (table test-table [id=bca0231a8b5f4f70861320493dd8d28e]) successfully deleted
I20260929 16:35:16.047677 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:16.020291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.17.205.1:33137)
0929 16:35:16.028435 (+ 8144us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:16.028442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:16.028469 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:35:16.028541 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:16.028560 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:16.028648 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:16.028778 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:16.033253 (+ 4475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.034627 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.036521 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.037147 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.037244 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:16.042486 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:16.042503 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:16.042520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:16.042555 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:16.045232 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.045956 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.045972 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.045975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.046119 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.046396 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.046399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.046401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.046725 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:16.046929 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:16.047202 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:35:16.047408 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8003,"thread_start_us":7858,"threads_started":1}
W20260929 16:35:16.048331 20177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59848
W20260929 16:35:16.048916 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20046: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:16.049126 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20046
I20260929 16:35:16.070595 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20186
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:16.085187 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:16.071366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.17.205.2:42855)
0929 16:35:16.071866 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:16.071878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:16.071892 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:16.071944 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:16.071947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:16.072006 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:16.072097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:16.073322 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.074069 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.075307 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.075724 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.075837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:16.079824 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:16.079847 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:16.079864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:16.079908 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:16.082458 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.083153 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.083161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.083164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.083332 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.083688 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.083693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.083695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.084062 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:16.084369 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:16.084636 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:35:16.084845 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":234,"threads_started":1}
W20260929 16:35:16.085808 20316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48398
W20260929 16:35:16.086381 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20186: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:16.086519 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20186
I20260929 16:35:16.100768 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20325
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[19930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:16.114722 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:16.101562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.17.205.3:43757)
0929 16:35:16.102016 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:16.102021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:16.102037 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:35:16.102090 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:16.102094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:16.102160 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:16.102243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:16.103529 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.104269 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.105416 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:16.105761 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:16.105840 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:16.109780 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:16.109795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:16.109811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:16.109849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:16.112475 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.113086 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.113094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.113096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.113209 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:16.113520 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:16.113523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:16.113525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:16.113850 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:16.114075 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:16.114317 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:35:16.114474 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
W20260929 16:35:16.115361 20454 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45762
W20260929 16:35:16.115906 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:16.116058 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20325
I20260929 16:35:16.133270 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 19943
I20260929 16:35:16.134465 20015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:16.248775 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 19943
2026-09-29T16:35:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8039 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[20619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[20619](info): set up 2 sockets
Sep 29 16:35:16 dist-test-slave-3h01 krb5kdc[20619](info): commencing operation
krb5kdc: starting...
W20260929 16:35:17.231153 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:17 dist-test-slave-3h01 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:17.415196 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43735
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:43735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:17.741138 20632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:17.741436 20632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:17.741536 20632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:17.748823 20632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:17.748955 20632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:17.749001 20632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:17.749058 20632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:17.749100 20632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:17.758924 20632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:43735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:17.760339 20632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:17.761620 20632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:17.773252 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:17.773290 20638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:17.773226 20640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:17.774212 20632 server_base.cc:1034] running on GCE node
I20260929 16:35:17.775038 20632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:17.776552 20632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:17.777760 20632 hybrid_clock.cc:648] HybridClock initialized: now 1790699717777701 us; error 49 us; skew 500 ppm
I20260929 16:35:17.778092 20632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:17 dist-test-slave-3h01 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699717, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:17.781123 20632 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:17.782625 20632 webserver.cc:460] Webserver started at http://127.17.205.62:46557/ using document root <none> and password file <none>
I20260929 16:35:17.783447 20632 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:17.783571 20632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:17.783871 20632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:17.786909 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "4628d97f20d74320b0d9b02caa5dc1ec"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-3h01"
server_key: "5e2d49b6ace62bcb328afc72ba06a813"
server_key_iv: "d0d47b4c2d5583c1d7e50794be688543"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.787695 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "4628d97f20d74320b0d9b02caa5dc1ec"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-3h01"
server_key: "5e2d49b6ace62bcb328afc72ba06a813"
server_key_iv: "d0d47b4c2d5583c1d7e50794be688543"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.792853 20632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:17.796327 20647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:17.797777 20632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:17.797981 20632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "4628d97f20d74320b0d9b02caa5dc1ec"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-3h01"
server_key: "5e2d49b6ace62bcb328afc72ba06a813"
server_key_iv: "d0d47b4c2d5583c1d7e50794be688543"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.798185 20632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:17.812052 20632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:17.816466 20632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:17.816763 20632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:17.834591 20632 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:43735
I20260929 16:35:17.834594 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:43735 every 8 connection(s)
I20260929 16:35:17.836138 20632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:17.839864 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:17.842439 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20632
I20260929 16:35:17.842635 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:17.843201 18228 external_mini_cluster.cc:1428] Setting key 7407639c86cc01e118a0d658902c8239
I20260929 16:35:17.851588 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec: Bootstrap starting.
I20260929 16:35:17.855283 20699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:17.856573 20699 log.cc:824] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:35:17 dist-test-slave-3h01 krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:17.859634 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec: No bootstrap required, opened a new log
I20260929 16:35:17.865681 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:17.845136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.17.205.62:43735)
0929 16:35:17.845871 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:17.845884 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:17.845925 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:35:17.847597 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:17.847629 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:17.848051 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:17.848385 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:17.850149 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.851459 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.853091 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.853782 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.853910 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:17.858550 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:17.858558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:17.858574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:17.858609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:17.861548 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.862350 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.862359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.862368 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.862574 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.862936 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.862939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.862941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.863553 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:17.863785 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:17.864198 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:35:17.864610 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":330,"threads_started":1}
I20260929 16:35:17.868268 20699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } }
I20260929 16:35:17.868650 20699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:17.868758 20699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4628d97f20d74320b0d9b02caa5dc1ec, State: Initialized, Role: FOLLOWER
I20260929 16:35:17.869318 20699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } }
I20260929 16:35:17.869521 20699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:17.869666 20699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:17.869822 20699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:17.872051 20699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } }
I20260929 16:35:17.872536 20699 leader_election.cc:302] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4628d97f20d74320b0d9b02caa5dc1ec; no voters:
I20260929 16:35:17.873070 20699 leader_election.cc:288] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:17.873183 20704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:17.874054 20704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [term 1 LEADER]: Becoming Leader. State: Replica: 4628d97f20d74320b0d9b02caa5dc1ec, State: Running, Role: LEADER
I20260929 16:35:17.874538 20704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } }
I20260929 16:35:17.874919 20699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:17.877206 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4628d97f20d74320b0d9b02caa5dc1ec. Latest consensus state: current_term: 1 leader_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } } }
I20260929 16:35:17.877182 20705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4628d97f20d74320b0d9b02caa5dc1ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43735 } } }
I20260929 16:35:17.877568 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:17.877624 20705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:17.879433 20710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:17.884552 20710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:17.892884 20710 catalog_manager.cc:1347] Generated new cluster ID: 7a93548ab6d64284ba03a369e90cd934
I20260929 16:35:17.893015 20710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:17.931037 20710 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:17.932428 20710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:17.940116 20710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4628d97f20d74320b0d9b02caa5dc1ec: Generated new TSK 0
I20260929 16:35:17.941378 20710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:17.989956 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:18.311769 20725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:18.312105 20725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:18.312234 20725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:18.319526 20725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:18.319651 20725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:18.319803 20725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:18.329869 20725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:18.331198 20725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:18.332592 20725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:18.345813 20730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:18.345831 20733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:18.345793 20731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:18.346743 20725 server_base.cc:1034] running on GCE node
I20260929 16:35:18.347509 20725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:18.348533 20725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:18.349753 20725 hybrid_clock.cc:648] HybridClock initialized: now 1790699718349688 us; error 59 us; skew 500 ppm
I20260929 16:35:18.350096 20725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:18 dist-test-slave-3h01 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:18.353295 20725 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:18.354835 20725 webserver.cc:460] Webserver started at http://127.17.205.1:44573/ using document root <none> and password file <none>
I20260929 16:35:18.355723 20725 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:18.355870 20725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:18.356180 20725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:18.359259 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "9257699a638f4b2a8cb7dadd2527d5cd"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "1df6f617399d703bff18e265c9153ad9"
server_key_iv: "f9c207c8d50f8ad21c0cf3822c4a374b"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.360078 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "9257699a638f4b2a8cb7dadd2527d5cd"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "1df6f617399d703bff18e265c9153ad9"
server_key_iv: "f9c207c8d50f8ad21c0cf3822c4a374b"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.365670 20725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:18.369356 20740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.370820 20725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:18.370999 20725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "9257699a638f4b2a8cb7dadd2527d5cd"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "1df6f617399d703bff18e265c9153ad9"
server_key_iv: "f9c207c8d50f8ad21c0cf3822c4a374b"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.371201 20725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:18.409850 20725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:18.414059 20725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:18.414341 20725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:18.415405 20725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:18.416770 20725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:18.416857 20725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.416980 20725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:18.417032 20725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.446817 20725 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:46727
I20260929 16:35:18.446861 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:46727 every 8 connection(s)
I20260929 16:35:18.448158 20725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:18.448557 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20725
I20260929 16:35:18.448791 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:18.449568 18228 external_mini_cluster.cc:1428] Setting key 37dcdc3d13b75a11d532c84fe33f10f3
Sep 29 16:35:18 dist-test-slave-3h01 krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:35:18.470435 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.451069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:35337 (local address 127.17.205.62:43735)
0929 16:35:18.451536 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:18.451542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:18.452726 (+ 1184us) server_negotiation.cc:407] Connection header received
0929 16:35:18.454547 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:18.454553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:18.454634 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:18.454726 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:18.456394 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.457124 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.459416 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.459848 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.459956 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.464315 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:18.464321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:18.464324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:18.464363 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:18.466909 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:18.467865 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:18.467870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:18.467873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:18.468011 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:18.468472 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:18.468477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:18.468479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:18.468783 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:18.468914 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:18.469882 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 16:35:18.470105 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
I20260929 16:35:18.471524 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.451818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43735 (local address 127.17.205.1:35337)
0929 16:35:18.452578 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:18.452625 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:18.454271 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:18.454982 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:18.454992 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:18.455570 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:18.456193 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:18.456221 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.457278 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.457283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:18.459260 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:18.459277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.460089 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.460095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:18.460395 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.461160 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:18.461189 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:18.463969 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:18.467110 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:18.467122 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:18.467146 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:18.467698 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:18.468123 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:18.468128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:18.468130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:18.468334 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:18.468974 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:18.468984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:18.469408 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 16:35:18.469722 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 16:35:18.470290 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260929 16:35:18.473156 20853 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43735
I20260929 16:35:18.473448 20853 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:18.474231 20853 heartbeater.cc:511] Master 127.17.205.62:43735 requested a full tablet report, sending...
I20260929 16:35:18.476691 20664 ts_manager.cc:194] Registered new tserver with Master: 9257699a638f4b2a8cb7dadd2527d5cd (127.17.205.1:46727)
I20260929 16:35:18.478618 20664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:35337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:18.494555 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:18.809662 20859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:18.810016 20859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:18.810125 20859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:18.817335 20859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:18.817464 20859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:18.817584 20859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:18.827378 20859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:18.828651 20859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:18.829952 20859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:18.843200 20865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:18.843432 20864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:18.843344 20867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:18.844259 20859 server_base.cc:1034] running on GCE node
I20260929 16:35:18.844925 20859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:18.845958 20859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:18.847172 20859 hybrid_clock.cc:648] HybridClock initialized: now 1790699718847111 us; error 59 us; skew 500 ppm
I20260929 16:35:18.847529 20859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:18 dist-test-slave-3h01 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:18.850749 20859 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:18.852360 20859 webserver.cc:460] Webserver started at http://127.17.205.2:42369/ using document root <none> and password file <none>
I20260929 16:35:18.853044 20859 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:18.853169 20859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:18.853453 20859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:18.856449 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "7bd8b84e3b7c42d4bdb873d7fba819f0"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "c2859f8233f298f5c70c533698bca9f6"
server_key_iv: "e65d22112dd67e651498e05dfc12747a"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.857209 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd8b84e3b7c42d4bdb873d7fba819f0"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "c2859f8233f298f5c70c533698bca9f6"
server_key_iv: "e65d22112dd67e651498e05dfc12747a"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.862543 20859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:18.866133 20874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.867669 20859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:35:18.867825 20859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "7bd8b84e3b7c42d4bdb873d7fba819f0"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-3h01"
server_key: "c2859f8233f298f5c70c533698bca9f6"
server_key_iv: "e65d22112dd67e651498e05dfc12747a"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.868045 20859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:18.914232 20859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:18.918409 20859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:18.918689 20859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:18.919679 20859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:18.921051 20859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:18.921133 20859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.921248 20859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:18.921299 20859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.950253 20859 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38805
I20260929 16:35:18.950304 20986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38805 every 8 connection(s)
I20260929 16:35:18.951573 20859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:18.952657 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20859
I20260929 16:35:18.952839 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:18.953331 18228 external_mini_cluster.cc:1428] Setting key e8afb5a819d8b2dfed26791cb29683dc
Sep 29 16:35:18 dist-test-slave-3h01 krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:35:18.971117 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.954245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:51137 (local address 127.17.205.62:43735)
0929 16:35:18.954409 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:18.954415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:18.955805 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 16:35:18.957598 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:18.957603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:18.957679 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:18.957774 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:18.959248 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.959938 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.961564 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.962166 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.962284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.965608 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:18.965613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:18.965616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:18.965649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:18.968207 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:18.968918 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:18.968924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:18.968928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:18.969054 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:18.969461 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:18.969466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:18.969470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:18.969837 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:18.969978 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:18.970677 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 16:35:18.970875 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:35:18.972843 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.954831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43735 (local address 127.17.205.2:51137)
0929 16:35:18.955650 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:18.955701 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:18.957333 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:18.957967 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:18.957976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:18.958509 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:18.959073 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:18.959092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.960092 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.960097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:18.961411 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:18.961423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.962339 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.962343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:18.962531 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.963142 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:18.963162 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:18.965380 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:18.968354 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:18.968364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:18.968381 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:18.968815 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:18.969159 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:18.969164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:18.969168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:18.969356 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:18.969974 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:18.969981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:18.970321 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 16:35:18.970566 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 16:35:18.970996 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":338,"threads_started":1}
I20260929 16:35:18.974303 20987 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43735
I20260929 16:35:18.974653 20987 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:18.975539 20987 heartbeater.cc:511] Master 127.17.205.62:43735 requested a full tablet report, sending...
I20260929 16:35:18.977273 20664 ts_manager.cc:194] Registered new tserver with Master: 7bd8b84e3b7c42d4bdb873d7fba819f0 (127.17.205.2:38805)
I20260929 16:35:18.978338 20664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:51137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:18.994954 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:19.318985 20992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:19.319334 20992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:19.319465 20992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:19.326961 20992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:19.327085 20992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:19.327257 20992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:19.337116 20992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:19.338475 20992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:19.339910 20992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:19.353740 21000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:19.353799 20998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:19.353834 20997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:19.354437 20992 server_base.cc:1034] running on GCE node
I20260929 16:35:19.355069 20992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:19.356129 20992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:19.357347 20992 hybrid_clock.cc:648] HybridClock initialized: now 1790699719357288 us; error 55 us; skew 500 ppm
I20260929 16:35:19.357688 20992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:19 dist-test-slave-3h01 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:19.360929 20992 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:19.362589 20992 webserver.cc:460] Webserver started at http://127.17.205.3:39985/ using document root <none> and password file <none>
I20260929 16:35:19.363355 20992 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:19.363476 20992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:19.363783 20992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:19.366842 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "1983f9e823634d969600b0c7ce90c2e1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-3h01"
server_key: "4a44de5cb226b8ce2435f1dd1daa5c63"
server_key_iv: "efe447b72dc4760fca5c57948fa7d57a"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.367667 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "1983f9e823634d969600b0c7ce90c2e1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-3h01"
server_key: "4a44de5cb226b8ce2435f1dd1daa5c63"
server_key_iv: "efe447b72dc4760fca5c57948fa7d57a"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.372867 20992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:19.376535 21007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:19.378003 20992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:19.378175 20992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "1983f9e823634d969600b0c7ce90c2e1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-3h01"
server_key: "4a44de5cb226b8ce2435f1dd1daa5c63"
server_key_iv: "efe447b72dc4760fca5c57948fa7d57a"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.378369 20992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:19.398850 20992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:19.403329 20992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:19.403597 20992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:19.404577 20992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:19.405916 20992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:19.405992 20992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:19.406117 20992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:19.406168 20992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:19.435828 20992 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:35497
I20260929 16:35:19.435874 21119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:35497 every 8 connection(s)
I20260929 16:35:19.437045 20992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:19.442473 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 20992
I20260929 16:35:19.442680 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:19.443194 18228 external_mini_cluster.cc:1428] Setting key 606ef476980c92e40e1fdbf737807649
Sep 29 16:35:19 dist-test-slave-3h01 krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:19.459403 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:19.439730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:50347 (local address 127.17.205.62:43735)
0929 16:35:19.439892 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:19.439896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:19.441062 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 16:35:19.443137 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:19.443142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:19.443236 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:19.443341 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:19.445254 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:19.446324 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:19.448669 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:19.449058 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:19.449156 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:19.453377 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:19.453383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:19.453386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:19.453422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:19.456006 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:19.456954 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:19.456959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:19.456962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:19.457116 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:19.457578 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:19.457582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:19.457584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:19.457936 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:19.458077 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:19.458806 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 16:35:19.459038 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:35:19.460484 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:19.440227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43735 (local address 127.17.205.3:50347)
0929 16:35:19.440902 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:19.440945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:19.442819 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:19.443614 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:19.443628 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:19.444287 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:19.445072 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:19.445099 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:19.446485 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:19.446491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:19.448476 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:19.448493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:19.449288 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:19.449295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:19.449590 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:19.450371 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:19.450407 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:19.453118 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:19.456180 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:19.456189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:19.456205 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:19.456798 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:19.457238 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:19.457243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:19.457247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:19.457440 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:19.458068 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:19.458077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:19.458435 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 16:35:19.458701 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 16:35:19.459293 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260929 16:35:19.461858 21120 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43735
I20260929 16:35:19.462127 21120 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:19.462817 21120 heartbeater.cc:511] Master 127.17.205.62:43735 requested a full tablet report, sending...
I20260929 16:35:19.464179 20664 ts_manager.cc:194] Registered new tserver with Master: 1983f9e823634d969600b0c7ce90c2e1 (127.17.205.3:35497)
I20260929 16:35:19.464929 20664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:50347
I20260929 16:35:19.471031 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:19.482107 20853 heartbeater.cc:503] Master 127.17.205.62:43735 was elected leader, sending a full tablet report...
W20260929 16:35:19.499845 20855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:19.497406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.17.205.62:43735)
0929 16:35:19.497667 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:19.497675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:19.498001 (+ 326us) server_negotiation.cc:407] Connection header received
0929 16:35:19.498194 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:19.498198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:19.498262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:19.498376 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:19.499656 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36658: BlockingRecv error: recv got EOF from 127.0.0.1:36658 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 16:35:19.502983 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20725
W20260929 16:35:19.505806 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20725: Not authorized: Client connection negotiation failed: client connection to 127.17.205.1:46727: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:35:19.505996 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20725
I20260929 16:35:19.518766 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20859
W20260929 16:35:19.521368 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20859: Not authorized: Client connection negotiation failed: client connection to 127.17.205.2:38805: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:35:19.521520 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20859
I20260929 16:35:19.533380 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20992
W20260929 16:35:19.535946 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20992: Not authorized: Client connection negotiation failed: client connection to 127.17.205.3:35497: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:35:19.536072 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20992
I20260929 16:35:19.548135 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 20632
I20260929 16:35:19.549279 20694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:19.649678 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 20632
2026-09-29T16:35:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3398 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:21 dist-test-slave-3h01 krb5kdc[21137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:21 dist-test-slave-3h01 krb5kdc[21137](info): set up 2 sockets
Sep 29 16:35:21 dist-test-slave-3h01 krb5kdc[21137](info): commencing operation
krb5kdc: starting...
W20260929 16:35:22.903741 18228 mini_kdc.cc:121] Time spent starting KDC: real 3.217s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 16:35:23.225293 18228 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.321s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 16:35:23.461365 18228 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.236s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 16:35:23.586092 18228 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.125s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/test-user.keytab.
W20260929 16:35:24.400689 18228 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.814s user 0.003s sys 0.003s
Sep 29 16:35:24 dist-test-slave-3h01 krb5kdc[21137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:24.486812 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43161
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:43161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:24.819607 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:24.819936 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:24.820081 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:24.827294 21150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:24.827430 21150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:24.827509 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:24.827582 21150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:24.827647 21150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:24.837496 21150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:43161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:24.838877 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:24.840303 21150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:24.851953 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:24.852017 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:24.851958 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:24.852711 21150 server_base.cc:1034] running on GCE node
I20260929 16:35:24.853547 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:24.855049 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:24.856259 21150 hybrid_clock.cc:648] HybridClock initialized: now 1790699724856198 us; error 55 us; skew 500 ppm
I20260929 16:35:24.856604 21150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:24 dist-test-slave-3h01 krb5kdc[21137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699724, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:24.864480 21150 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:24.866015 21150 webserver.cc:460] Webserver started at http://127.17.205.62:38787/ using document root <none> and password file <none>
I20260929 16:35:24.866761 21150 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:24.866884 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:24.867182 21150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:24.870340 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "a38bdd5f8c2e43298aa4cac0dfda0171"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-3h01"
server_key: "d9f61780cbeb14bc5cebf45778ff2102"
server_key_iv: "aa401a0d0ad498c74d8d52d010ce7642"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.871129 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "a38bdd5f8c2e43298aa4cac0dfda0171"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-3h01"
server_key: "d9f61780cbeb14bc5cebf45778ff2102"
server_key_iv: "aa401a0d0ad498c74d8d52d010ce7642"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.876407 21150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:35:24.879889 21165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:24.881330 21150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:24.881502 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a38bdd5f8c2e43298aa4cac0dfda0171"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-3h01"
server_key: "d9f61780cbeb14bc5cebf45778ff2102"
server_key_iv: "aa401a0d0ad498c74d8d52d010ce7642"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.881693 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:24.893188 21150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:24.897058 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:24.897332 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:24.913285 21150 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:43161
I20260929 16:35:24.913311 21216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:43161 every 8 connection(s)
I20260929 16:35:24.914685 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:24.918672 21217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:24.924294 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21150
I20260929 16:35:24.924503 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:24.925122 18228 external_mini_cluster.cc:1428] Setting key f3dc3daae1c13e9676c1de7d52d50b28
I20260929 16:35:24.930610 21217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171: Bootstrap starting.
I20260929 16:35:24.934530 21217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:24.935675 21217 log.cc:824] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:24.938657 21217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171: No bootstrap required, opened a new log
Sep 29 16:35:24 dist-test-slave-3h01 krb5kdc[21137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:24.948777 21217 raft_consensus.cc:348] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } }
I20260929 16:35:24.949244 21217 raft_consensus.cc:374] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:24.949350 21217 raft_consensus.cc:729] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38bdd5f8c2e43298aa4cac0dfda0171, State: Initialized, Role: FOLLOWER
I20260929 16:35:24.949851 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:24.927105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.17.205.62:43161)
0929 16:35:24.928005 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:24.928022 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:24.928075 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:35:24.930399 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:24.930430 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:24.930970 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:24.931470 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:24.932782 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:24.934685 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:24.936022 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:24.936854 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:24.937031 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:24.940955 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:24.940966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:24.940987 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:24.941040 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:24.945106 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:24.945823 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:24.945833 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:24.945844 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:24.946058 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:24.946427 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:24.946432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:24.946435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:24.947272 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:24.947558 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:24.947944 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 16:35:24.948409 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":327,"threads_started":1}
I20260929 16:35:24.950059 21217 consensus_queue.cc:260] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } }
I20260929 16:35:24.950327 21217 raft_consensus.cc:388] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:24.950477 21217 raft_consensus.cc:482] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:24.950685 21217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:24.953354 21217 raft_consensus.cc:504] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } }
I20260929 16:35:24.953938 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:24.953933 21217 leader_election.cc:302] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a38bdd5f8c2e43298aa4cac0dfda0171; no voters:
I20260929 16:35:24.954537 21217 leader_election.cc:288] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:24.954659 21222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:24.956619 21217 sys_catalog.cc:564] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:24.962376 21222 raft_consensus.cc:686] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [term 1 LEADER]: Becoming Leader. State: Replica: a38bdd5f8c2e43298aa4cac0dfda0171, State: Running, Role: LEADER
I20260929 16:35:24.963104 21222 consensus_queue.cc:237] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } }
I20260929 16:35:24.966073 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } } }
I20260929 16:35:24.966449 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:24.966938 21224 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a38bdd5f8c2e43298aa4cac0dfda0171. Latest consensus state: current_term: 1 leader_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38bdd5f8c2e43298aa4cac0dfda0171" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43161 } } }
I20260929 16:35:24.967188 21224 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:24.975584 21233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:24.981945 21233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:24.991357 21233 catalog_manager.cc:1347] Generated new cluster ID: 7eb2fc7cefad41ffa5f7f9a24bca803d
I20260929 16:35:24.991536 21233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:25.005837 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:24.998714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.17.205.62:43161)
0929 16:35:24.998862 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:24.998867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:24.999469 (+ 602us) server_negotiation.cc:407] Connection header received
0929 16:35:24.999644 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:24.999649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:24.999719 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:24.999799 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.000846 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.001576 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.002830 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.003326 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.003439 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.004457 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.004461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.004465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:25.005092 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.005168 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:25.005192 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.005283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.005411 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.005504 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:35:25.018098 21233 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:25.019661 21233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:25.034591 21233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a38bdd5f8c2e43298aa4cac0dfda0171: Generated new TSK 0
I20260929 16:35:25.035516 21233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:25.044226 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21150
I20260929 16:35:25.045567 21212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:25.135097 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21150
2026-09-29T16:35:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (5480 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:25 dist-test-slave-3h01 krb5kdc[21251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:25 dist-test-slave-3h01 krb5kdc[21251](info): set up 2 sockets
Sep 29 16:35:25 dist-test-slave-3h01 krb5kdc[21251](info): commencing operation
krb5kdc: starting...
W20260929 16:35:26.119379 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:26 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:26.302591 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41817
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:26.621660 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:26.621959 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:26.622052 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:26.629195 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:26.629315 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:26.629367 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:26.629421 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:26.629464 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:26.639302 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:26.640622 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:26.641954 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:26.653507 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:26.653431 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:26.653452 21272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:26.654476 21264 server_base.cc:1034] running on GCE node
I20260929 16:35:26.655275 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:26.656697 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:26.657896 21264 hybrid_clock.cc:648] HybridClock initialized: now 1790699726657842 us; error 48 us; skew 500 ppm
I20260929 16:35:26.658233 21264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:26 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699726, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:26.661422 21264 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:26.662861 21264 webserver.cc:460] Webserver started at http://127.17.205.62:40257/ using document root <none> and password file <none>
I20260929 16:35:26.663585 21264 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:26.663700 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:26.663992 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:26.666960 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "dab8bf5aa66542828f17d09d37966cf2"
format_stamp: "Formatted at 2026-09-29 16:35:26 on dist-test-slave-3h01"
server_key: "668683499b9385f6263af4696875f36b"
server_key_iv: "f20cebb77f9cb077cf40696e0e6c737d"
server_key_version: "encryptionkey@0"
I20260929 16:35:26.667726 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "dab8bf5aa66542828f17d09d37966cf2"
format_stamp: "Formatted at 2026-09-29 16:35:26 on dist-test-slave-3h01"
server_key: "668683499b9385f6263af4696875f36b"
server_key_iv: "f20cebb77f9cb077cf40696e0e6c737d"
server_key_version: "encryptionkey@0"
I20260929 16:35:26.672911 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:26.676514 21279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:26.677862 21264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:26.678026 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "dab8bf5aa66542828f17d09d37966cf2"
format_stamp: "Formatted at 2026-09-29 16:35:26 on dist-test-slave-3h01"
server_key: "668683499b9385f6263af4696875f36b"
server_key_iv: "f20cebb77f9cb077cf40696e0e6c737d"
server_key_version: "encryptionkey@0"
I20260929 16:35:26.678203 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:26.696767 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:26.701349 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:26.701623 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:26.718194 21264 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:41817
I20260929 16:35:26.718212 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:41817 every 8 connection(s)
I20260929 16:35:26.719647 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:26.720013 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21264
I20260929 16:35:26.720213 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:26.720993 18228 external_mini_cluster.cc:1428] Setting key 4caca963b1b9afdc0c10de43425fd941
I20260929 16:35:26.724196 21331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:35:26 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:26.737022 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2: Bootstrap starting.
I20260929 16:35:26.741160 21331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:26.742487 21331 log.cc:824] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:26.744700 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:26.722928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.17.205.62:41817)
0929 16:35:26.723697 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.723714 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.723765 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:35:26.725932 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.725963 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.726478 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.726928 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.728190 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:26.730192 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:26.731484 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:26.732483 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:26.732669 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:26.736557 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:26.736568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:26.736591 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:26.736640 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:26.740261 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:26.740964 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:26.740974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:26.740983 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:26.741193 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:26.741585 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:26.741590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:26.741593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:26.742290 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:26.742579 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:26.742993 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:35:26.743462 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":320,"threads_started":1}
I20260929 16:35:26.745730 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2: No bootstrap required, opened a new log
I20260929 16:35:26.753320 21331 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } }
I20260929 16:35:26.753701 21331 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:26.753798 21331 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab8bf5aa66542828f17d09d37966cf2, State: Initialized, Role: FOLLOWER
I20260929 16:35:26.754344 21331 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } }
I20260929 16:35:26.754559 21331 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:26.754699 21331 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:26.754865 21331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:26.757007 21331 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } }
I20260929 16:35:26.757510 21331 leader_election.cc:302] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab8bf5aa66542828f17d09d37966cf2; no voters:
I20260929 16:35:26.758047 21331 leader_election.cc:288] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:26.758160 21336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:26.759088 21336 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [term 1 LEADER]: Becoming Leader. State: Replica: dab8bf5aa66542828f17d09d37966cf2, State: Running, Role: LEADER
I20260929 16:35:26.759724 21336 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } }
I20260929 16:35:26.760170 21331 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:26.762401 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab8bf5aa66542828f17d09d37966cf2. Latest consensus state: current_term: 1 leader_uuid: "dab8bf5aa66542828f17d09d37966cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } } }
I20260929 16:35:26.762812 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:26.763319 21337 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab8bf5aa66542828f17d09d37966cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab8bf5aa66542828f17d09d37966cf2" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41817 } } }
I20260929 16:35:26.763618 21337 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:26.763729 21345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:26.769656 21345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:26.778406 21345 catalog_manager.cc:1347] Generated new cluster ID: f80b2b8314f44971a6efa0e14fbea839
I20260929 16:35:26.778532 21345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:26.795671 21345 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:26.797595 21345 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:26.805868 21345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dab8bf5aa66542828f17d09d37966cf2: Generated new TSK 0
I20260929 16:35:26.806735 21345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:26.860172 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:27.186189 21357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:27.186451 21357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:27.186576 21357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:27.193675 21357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:27.193779 21357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:27.193930 21357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:27.203501 21357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:27.204717 21357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:27.206032 21357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:27.219336 21362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:27.219365 21365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:27.219775 21363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:27.220230 21357 server_base.cc:1034] running on GCE node
I20260929 16:35:27.220899 21357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:27.221850 21357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:27.223078 21357 hybrid_clock.cc:648] HybridClock initialized: now 1790699727223011 us; error 54 us; skew 500 ppm
I20260929 16:35:27.223421 21357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:27 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699727, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:27.226372 21357 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:27.227922 21357 webserver.cc:460] Webserver started at http://127.17.205.1:36411/ using document root <none> and password file <none>
I20260929 16:35:27.228611 21357 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:27.228732 21357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:27.229013 21357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:27.231966 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "cb2a990271e1495a80d22d1076ccff95"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "4988794fc0a6c501277b7c07489e21ec"
server_key_iv: "e78083b06b53601bfdbe6129a0ae0e2f"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.232735 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "cb2a990271e1495a80d22d1076ccff95"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "4988794fc0a6c501277b7c07489e21ec"
server_key_iv: "e78083b06b53601bfdbe6129a0ae0e2f"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.237856 21357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:35:27.241400 21372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.242815 21357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:35:27.242970 21357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "cb2a990271e1495a80d22d1076ccff95"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "4988794fc0a6c501277b7c07489e21ec"
server_key_iv: "e78083b06b53601bfdbe6129a0ae0e2f"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.243148 21357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:27.257990 21357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:27.261958 21357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:27.262238 21357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:27.263266 21357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:27.264667 21357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:27.264742 21357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.264854 21357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:27.264907 21357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.294049 21357 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:37365
I20260929 16:35:27.294090 21484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:37365 every 8 connection(s)
I20260929 16:35:27.295358 21357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:27.300333 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21357
I20260929 16:35:27.300536 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:27.301306 18228 external_mini_cluster.cc:1428] Setting key 63a25365ea8cef2b0d51562d62b40bc6
Sep 29 16:35:27 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699727, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:27.316287 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.297819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:59581 (local address 127.17.205.62:41817)
0929 16:35:27.298215 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.298220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.299109 (+ 889us) server_negotiation.cc:407] Connection header received
0929 16:35:27.301429 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.301434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.301511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.301602 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:27.303333 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.304151 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.306331 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.306740 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.306845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.310710 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:27.310715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:27.310719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:27.310758 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:27.313128 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:27.313981 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:27.313984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:27.313987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:27.314099 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:27.314535 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:27.314537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:27.314539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:27.314814 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:27.314917 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.315832 (+ 915us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.316002 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":207,"threads_started":1}
I20260929 16:35:27.317438 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.298296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41817 (local address 127.17.205.1:59581)
0929 16:35:27.298955 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:27.298997 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:27.301098 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:27.301877 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:27.301890 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:27.302496 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:27.303145 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:27.303171 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.304315 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.304321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:27.306177 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:27.306191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.306953 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.306960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:27.307242 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.307924 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:27.307951 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:27.310468 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:27.313277 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:27.313286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:27.313304 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:27.313877 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:27.314188 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:27.314193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:27.314197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:27.314442 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:27.314931 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:27.314939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:27.315413 (+ 474us) client_negotiation.cc:769] Sending connection context
0929 16:35:27.315736 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 16:35:27.316303 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":248,"threads_started":1}
I20260929 16:35:27.319031 21485 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41817
I20260929 16:35:27.319382 21485 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:35:27.320142 21485 heartbeater.cc:511] Master 127.17.205.62:41817 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:35:27.322417 21296 ts_manager.cc:194] Registered new tserver with Master: cb2a990271e1495a80d22d1076ccff95 (127.17.205.1:37365)
I20260929 16:35:27.323943 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:59581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:27.344019 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:27.660535 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:27.660826 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:27.660924 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:27.668141 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:27.668247 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:27.668458 21491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:27.679713 21491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:27.681085 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:27.682504 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:27.695372 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:27.695317 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:27.695353 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:27.696542 21491 server_base.cc:1034] running on GCE node
I20260929 16:35:27.697116 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:27.698050 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:27.699276 21491 hybrid_clock.cc:648] HybridClock initialized: now 1790699727699216 us; error 46 us; skew 500 ppm
I20260929 16:35:27.699580 21491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:27 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699727, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:27.702559 21491 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:27.704221 21491 webserver.cc:460] Webserver started at http://127.17.205.2:34067/ using document root <none> and password file <none>
I20260929 16:35:27.704879 21491 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:27.704979 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:27.705229 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:27.708120 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "e79d8171d13d49a6aba29f93331c5096"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "d51c456dabb081d8546a54e32909a19e"
server_key_iv: "ffa951f99d753d05ffff548fbf033c9b"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.708856 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "e79d8171d13d49a6aba29f93331c5096"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "d51c456dabb081d8546a54e32909a19e"
server_key_iv: "ffa951f99d753d05ffff548fbf033c9b"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.713866 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:27.717253 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.718971 21491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:27.719142 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "e79d8171d13d49a6aba29f93331c5096"
format_stamp: "Formatted at 2026-09-29 16:35:27 on dist-test-slave-3h01"
server_key: "d51c456dabb081d8546a54e32909a19e"
server_key_iv: "ffa951f99d753d05ffff548fbf033c9b"
server_key_version: "encryptionkey@0"
I20260929 16:35:27.719386 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:27.734983 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:27.739789 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:27.740034 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:27.741003 21491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:27.742283 21491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:27.742357 21491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.742445 21491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:27.742486 21491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:27.772346 21491 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:43621
I20260929 16:35:27.772415 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:43621 every 8 connection(s)
I20260929 16:35:27.773581 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:27.781972 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21491
I20260929 16:35:27.782157 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:27.782747 18228 external_mini_cluster.cc:1428] Setting key ff366f47819aabf27e407ec903238bb4
Sep 29 16:35:27 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699727, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:27.793883 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.776164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57591 (local address 127.17.205.62:41817)
0929 16:35:27.776321 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.776325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.777483 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 16:35:27.779203 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.779239 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.779340 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.779430 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:27.780913 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.781642 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.783746 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.784189 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.784306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.788107 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:27.788112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:27.788115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:27.788149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:27.790856 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:27.791698 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:27.791702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:27.791705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:27.791835 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:27.792278 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:27.792281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:27.792283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:27.792535 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:27.792640 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.793486 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.793657 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:35:27.795288 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.776637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41817 (local address 127.17.205.2:57591)
0929 16:35:27.777315 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:27.777353 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:27.778919 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:27.779633 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:27.779641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:27.780143 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:27.780724 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:27.780743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.781800 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.781805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:27.783606 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:27.783622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.784402 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.784409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:27.784684 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.785379 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:27.785409 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:27.787889 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:27.790986 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:27.790995 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:27.791014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:27.791581 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:27.791934 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:27.791939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:27.791943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:27.792181 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:27.792673 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:27.792681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:27.793115 (+ 434us) client_negotiation.cc:769] Sending connection context
0929 16:35:27.793417 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 16:35:27.793930 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
I20260929 16:35:27.796736 21619 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41817
I20260929 16:35:27.797094 21619 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:27.798007 21619 heartbeater.cc:511] Master 127.17.205.62:41817 requested a full tablet report, sending...
I20260929 16:35:27.799660 21296 ts_manager.cc:194] Registered new tserver with Master: e79d8171d13d49a6aba29f93331c5096 (127.17.205.2:43621)
I20260929 16:35:27.800487 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:57591
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:27.824529 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:28.136821 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:28.137118 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:28.137260 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:28.144534 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:28.144661 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:28.144802 21624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:28.154613 21624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:28.155974 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:28.157277 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:28.170518 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.170487 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.170557 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:28.171080 21624 server_base.cc:1034] running on GCE node
I20260929 16:35:28.171749 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:28.172744 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:28.173955 21624 hybrid_clock.cc:648] HybridClock initialized: now 1790699728173901 us; error 45 us; skew 500 ppm
I20260929 16:35:28.174270 21624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:28.177368 21624 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:28.178784 21624 webserver.cc:460] Webserver started at http://127.17.205.3:39151/ using document root <none> and password file <none>
I20260929 16:35:28.179538 21624 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:28.179661 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:28.179946 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:28.182940 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "f09ec91c051e4c7698b20f6ca271bcaf"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-3h01"
server_key: "aa1dc51282f6b047f792d0b724252b7e"
server_key_iv: "1aaf680f87949a0769bba0c98c46d8f8"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.183776 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "f09ec91c051e4c7698b20f6ca271bcaf"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-3h01"
server_key: "aa1dc51282f6b047f792d0b724252b7e"
server_key_iv: "1aaf680f87949a0769bba0c98c46d8f8"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.188885 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:28.192363 21639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.193724 21624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:28.193895 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "f09ec91c051e4c7698b20f6ca271bcaf"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-3h01"
server_key: "aa1dc51282f6b047f792d0b724252b7e"
server_key_iv: "1aaf680f87949a0769bba0c98c46d8f8"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.194074 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:28.222303 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:28.226445 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:28.226722 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:28.227684 21624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:28.228929 21624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:28.228999 21624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.229110 21624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:28.229156 21624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.256834 21624 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:41363
I20260929 16:35:28.256875 21751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:41363 every 8 connection(s)
I20260929 16:35:28.258044 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:28.261058 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21624
I20260929 16:35:28.261206 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:28.261698 18228 external_mini_cluster.cc:1428] Setting key 8037ef38a8dc9a6dddb8fa9d0e0f0154
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:28.277534 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.260578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:49673 (local address 127.17.205.62:41817)
0929 16:35:28.260818 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.260824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.262222 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 16:35:28.264468 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.264475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.264569 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.264673 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.266232 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.266970 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.268491 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.268876 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.268964 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.272130 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.272135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.272138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.272178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.274422 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.275115 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.275119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.275121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.275279 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.275713 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.275716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.275718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.276046 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.276185 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.276956 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.277184 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:35:28.278471 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.261200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41817 (local address 127.17.205.3:49673)
0929 16:35:28.262039 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.262089 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.264179 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.264901 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.264914 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.265495 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.266082 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.266100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.267093 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.267097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.268377 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.268389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.269039 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.269044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.269232 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.269813 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.269834 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.271927 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.274551 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.274560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.274577 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.274991 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.275391 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.275395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.275398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.275593 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.276156 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.276165 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.276554 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.276851 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.277391 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":304,"threads_started":1}
I20260929 16:35:28.279740 21752 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41817
I20260929 16:35:28.280017 21752 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:28.280725 21752 heartbeater.cc:511] Master 127.17.205.62:41817 requested a full tablet report, sending...
I20260929 16:35:28.282069 21296 ts_manager.cc:194] Registered new tserver with Master: f09ec91c051e4c7698b20f6ca271bcaf (127.17.205.3:41363)
I20260929 16:35:28.282856 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:49673
I20260929 16:35:28.289551 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260929 16:35:28.326906 21485 heartbeater.cc:503] Master 127.17.205.62:41817 was elected leader, sending a full tablet report...
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:28.333781 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.320656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.17.205.62:41817)
0929 16:35:28.320827 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.320832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.321193 (+ 361us) server_negotiation.cc:407] Connection header received
0929 16:35:28.321381 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.321387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.321455 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.321545 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.322790 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.323568 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.324808 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.325225 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.325322 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.329026 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.329031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.329035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.329066 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.331255 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.331899 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.331904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.331908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.332037 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.332376 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.332380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.332382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.332709 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.332932 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.333263 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.333462 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:35:28.334359 21296 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41380
I20260929 16:35:28.336028 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21357
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:28.350827 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.337031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.17.205.1:37365)
0929 16:35:28.337590 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.337597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.337622 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:35:28.337677 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.337684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.337893 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.338013 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.339066 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.340093 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.341250 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.341805 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.341899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.345641 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.345655 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.345671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.345709 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.348572 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.349196 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.349202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.349204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.349315 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.349573 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.349576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.349578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.349916 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.350128 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.350346 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.350532 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
W20260929 16:35:28.352089 21480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60062
W20260929 16:35:28.353143 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:28.353332 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21357
I20260929 16:35:28.365796 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21491
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:28.380463 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.366589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.17.205.2:43621)
0929 16:35:28.367114 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.367122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.367147 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:35:28.367241 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.367251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.367486 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.367645 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.368766 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.369760 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.370869 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.371525 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.371623 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.375279 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.375292 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.375308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.375342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.377768 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.378392 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.378400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.378402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.378523 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.378868 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.378872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.378891 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.379266 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.379542 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.379860 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.380111 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
W20260929 16:35:28.381755 21614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59386
W20260929 16:35:28.382905 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:28.383052 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21491
I20260929 16:35:28.395049 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21624
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:28.410907 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.396010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.17.205.3:41363)
0929 16:35:28.396589 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.396597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.396632 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:35:28.396690 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.396697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.396905 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.397028 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.398379 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.399551 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.400749 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.401311 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.401403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.405419 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.405431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.405447 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.405482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.408439 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.409068 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.409074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.409077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.409204 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.409516 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.409520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.409522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.409909 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.410149 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.410354 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.410546 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":290,"threads_started":1}
W20260929 16:35:28.412233 21747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39012
W20260929 16:35:28.413270 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21624: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:35:28.413417 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21624
I20260929 16:35:28.425567 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21264
I20260929 16:35:28.426770 21326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:28.526775 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21264
2026-09-29T16:35:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3396 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21769](info): set up 2 sockets
Sep 29 16:35:28 dist-test-slave-3h01 krb5kdc[21769](info): commencing operation
krb5kdc: starting...
W20260929 16:35:29.519475 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:29 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:29.706992 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40785
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:40785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:30.023597 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:30.023893 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:30.023991 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:30.031787 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:30.031919 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:30.031963 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:30.032017 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:30.032063 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:30.042164 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:40785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:30.043545 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:30.044838 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:30.056072 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:30.056100 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:30.056133 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:30.056949 21782 server_base.cc:1034] running on GCE node
I20260929 16:35:30.057801 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:30.059278 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:30.060479 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790699730060430 us; error 45 us; skew 500 ppm
I20260929 16:35:30.060829 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:30 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699730, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:30.069988 21782 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:30.071650 21782 webserver.cc:460] Webserver started at http://127.17.205.62:40177/ using document root <none> and password file <none>
I20260929 16:35:30.072357 21782 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:30.072471 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:30.072765 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:30.075770 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "802e6d6c980f5fda101c04a646f91b16"
server_key_iv: "49840f8bc03e7d72acbe233939ee09a2"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.076529 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "802e6d6c980f5fda101c04a646f91b16"
server_key_iv: "49840f8bc03e7d72acbe233939ee09a2"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.081532 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:35:30.084888 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:30.086127 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:35:30.086284 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "802e6d6c980f5fda101c04a646f91b16"
server_key_iv: "49840f8bc03e7d72acbe233939ee09a2"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.086452 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:30.120927 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:30.125402 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:30.125665 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:30.142586 21782 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:40785
I20260929 16:35:30.142598 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:40785 every 8 connection(s)
I20260929 16:35:30.143967 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:30.145368 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21782
I20260929 16:35:30.145586 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:30.146330 18228 external_mini_cluster.cc:1428] Setting key aa044746b22575f03a362e8c6cd3313c
I20260929 16:35:30.148660 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:35:30 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:30.161389 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7: Bootstrap starting.
I20260929 16:35:30.165508 21849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:30.166793 21849 log.cc:824] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:30.169211 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.148391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.17.205.62:40785)
0929 16:35:30.149095 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.149108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.149151 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:35:30.150724 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.150745 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.151135 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.151521 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:30.153250 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.154584 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.156260 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.156917 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.157041 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.161470 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:30.161482 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:30.161503 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:30.161556 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:30.165175 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:30.165876 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:30.165886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:30.165896 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:30.166097 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:30.166401 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:30.166405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:30.166408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:30.167098 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:30.167432 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.167817 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.168220 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":308,"threads_started":1}
I20260929 16:35:30.169783 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7: No bootstrap required, opened a new log
I20260929 16:35:30.177466 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } }
I20260929 16:35:30.177835 21849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:30.177924 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4243ba8fc5ba4ff19fee9aaaf3a5c6f7, State: Initialized, Role: FOLLOWER
I20260929 16:35:30.178486 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } }
I20260929 16:35:30.178692 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:30.178815 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:30.178957 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:30.181020 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } }
I20260929 16:35:30.181495 21849 leader_election.cc:302] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4243ba8fc5ba4ff19fee9aaaf3a5c6f7; no voters:
I20260929 16:35:30.182042 21849 leader_election.cc:288] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:30.182161 21854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:30.182740 21854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4243ba8fc5ba4ff19fee9aaaf3a5c6f7, State: Running, Role: LEADER
I20260929 16:35:30.183348 21854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } }
I20260929 16:35:30.184170 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:30.186004 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4243ba8fc5ba4ff19fee9aaaf3a5c6f7. Latest consensus state: current_term: 1 leader_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } } }
I20260929 16:35:30.186414 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:30.188367 21862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:30.188169 21855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4243ba8fc5ba4ff19fee9aaaf3a5c6f7" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40785 } } }
I20260929 16:35:30.188594 21855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:30.193528 21862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:30.202484 21862 catalog_manager.cc:1347] Generated new cluster ID: c149364dd4bb43dbbee822774fc79455
I20260929 16:35:30.202612 21862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:30.226401 21862 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:30.227833 21862 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:30.242621 21862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4243ba8fc5ba4ff19fee9aaaf3a5c6f7: Generated new TSK 0
I20260929 16:35:30.243728 21862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:30.302389 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:30.634670 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:30.634954 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:30.635077 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:30.642247 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:30.642371 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:30.642521 21875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:30.652218 21875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:30.653498 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:30.654808 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:30.667752 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:30.667802 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:30.667706 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:30.668844 21875 server_base.cc:1034] running on GCE node
I20260929 16:35:30.669459 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:30.670445 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:30.671687 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790699730671630 us; error 45 us; skew 500 ppm
I20260929 16:35:30.672031 21875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:30 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699730, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:30.675477 21875 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:30.676954 21875 webserver.cc:460] Webserver started at http://127.17.205.1:40209/ using document root <none> and password file <none>
I20260929 16:35:30.677626 21875 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:30.677740 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:30.678035 21875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:30.681011 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "06f47e108ac5489a89a745f788f268ee"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "b3153122594e0b37217ab5542abb9954"
server_key_iv: "7cea05b1997e1136362b7021478bba7f"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.681818 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "06f47e108ac5489a89a745f788f268ee"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "b3153122594e0b37217ab5542abb9954"
server_key_iv: "7cea05b1997e1136362b7021478bba7f"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.687147 21875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:35:30.690908 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:30.692382 21875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:30.692536 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "06f47e108ac5489a89a745f788f268ee"
format_stamp: "Formatted at 2026-09-29 16:35:30 on dist-test-slave-3h01"
server_key: "b3153122594e0b37217ab5542abb9954"
server_key_iv: "7cea05b1997e1136362b7021478bba7f"
server_key_version: "encryptionkey@0"
I20260929 16:35:30.692737 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:30.712154 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:30.716171 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:30.716436 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:30.717397 21875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:30.718727 21875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:30.718819 21875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:30.718933 21875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:30.718986 21875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:30.748862 21875 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:41647
I20260929 16:35:30.748901 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:41647 every 8 connection(s)
I20260929 16:35:30.750134 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:30.752666 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 21875
I20260929 16:35:30.752883 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:30.753648 18228 external_mini_cluster.cc:1428] Setting key 993f1b087364211d0b509f7e0091b37e
Sep 29 16:35:30 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699730, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:35:30.771467 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.752836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:48591 (local address 127.17.205.62:40785)
0929 16:35:30.755542 (+ 2706us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.755546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.755562 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:35:30.756891 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.756897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.756973 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.757083 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:30.758621 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.759601 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.761240 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.761795 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.761923 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.765659 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:30.765664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:30.765667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:30.765699 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:30.768309 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:30.769169 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:30.769175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:30.769178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:30.769311 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:30.769717 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:30.769722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:30.769726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:30.769995 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:30.770121 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.770846 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.771072 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":2507,"threads_started":1}
I20260929 16:35:30.772550 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.753457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40785 (local address 127.17.205.1:48591)
0929 16:35:30.754444 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:30.754516 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:30.756597 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:30.757289 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:30.757297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:30.757857 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:30.758455 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:30.758475 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.759758 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.759763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:30.761093 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:30.761105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.762031 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.762038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:30.762320 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.763016 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:30.763043 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:30.765436 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:30.768469 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:30.768479 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:30.768499 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:30.769050 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:30.769413 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:30.769418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:30.769422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:30.769613 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:30.770114 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:30.770123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:30.770486 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 16:35:30.770736 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 16:35:30.771333 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":359,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:35:30.774214 22003 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40785
I20260929 16:35:30.774573 22003 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:30.775583 22003 heartbeater.cc:511] Master 127.17.205.62:40785 requested a full tablet report, sending...
I20260929 16:35:30.777911 21814 ts_manager.cc:194] Registered new tserver with Master: 06f47e108ac5489a89a745f788f268ee (127.17.205.1:41647)
I20260929 16:35:30.779376 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:48591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:30.798300 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:31.112861 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:31.113142 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:31.113281 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:31.120414 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:31.120530 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:31.120651 22009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:31.130443 22009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:31.131709 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:31.132949 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:31.145644 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:31.145669 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:31.146524 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:31.146759 22009 server_base.cc:1034] running on GCE node
I20260929 16:35:31.147403 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:31.148389 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:31.149601 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790699731149542 us; error 56 us; skew 500 ppm
I20260929 16:35:31.149910 22009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:31.153182 22009 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:31.154876 22009 webserver.cc:460] Webserver started at http://127.17.205.2:46517/ using document root <none> and password file <none>
I20260929 16:35:31.155786 22009 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:31.155920 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:31.156247 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:31.160159 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "f3259cc19a924cc0a8e86661088aa071"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "a957732bb95752b28b02cc2e524baac6"
server_key_iv: "5c2652933407b37185524afb73f898c6"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.160936 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "f3259cc19a924cc0a8e86661088aa071"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "a957732bb95752b28b02cc2e524baac6"
server_key_iv: "5c2652933407b37185524afb73f898c6"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.166016 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:31.169404 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.170764 22009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:31.170900 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "f3259cc19a924cc0a8e86661088aa071"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "a957732bb95752b28b02cc2e524baac6"
server_key_iv: "5c2652933407b37185524afb73f898c6"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.171063 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:31.190526 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:31.194916 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:31.195204 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:31.196183 22009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:31.197504 22009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:31.197578 22009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.197710 22009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:31.197759 22009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.225237 22009 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:36863
I20260929 16:35:31.225292 22136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:36863 every 8 connection(s)
I20260929 16:35:31.226678 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:31.236335 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22009
I20260929 16:35:31.236505 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:31.236912 18228 external_mini_cluster.cc:1428] Setting key 837d5901937d7898a128e604786180ec
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:31.246327 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.229267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:51467 (local address 127.17.205.62:40785)
0929 16:35:31.229442 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.229447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.230699 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 16:35:31.232545 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.232551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.232647 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.232730 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.234272 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.234994 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.236572 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.237008 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.237114 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.240786 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.240791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.240794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.240827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.243182 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.244059 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.244063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.244065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.244206 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.244665 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.244668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.244670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.244945 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.245049 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.245890 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.246064 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:35:31.247707 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.229746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40785 (local address 127.17.205.2:51467)
0929 16:35:31.230493 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:31.230551 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:31.232269 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:31.232988 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:31.232996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:31.233512 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:31.234081 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:31.234098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.235143 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.235147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:31.236438 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:31.236453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.237232 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.237240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:31.237487 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.238180 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:31.238207 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:31.240570 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:31.243331 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:31.243341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:31.243361 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:31.243912 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:31.244303 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:31.244308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:31.244311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:31.244557 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:31.245060 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:31.245069 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:31.245503 (+ 434us) client_negotiation.cc:769] Sending connection context
0929 16:35:31.245790 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 16:35:31.246326 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":299,"threads_started":1}
I20260929 16:35:31.249058 22137 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40785
I20260929 16:35:31.249372 22137 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:31.250221 22137 heartbeater.cc:511] Master 127.17.205.62:40785 requested a full tablet report, sending...
I20260929 16:35:31.251854 21814 ts_manager.cc:194] Registered new tserver with Master: f3259cc19a924cc0a8e86661088aa071 (127.17.205.2:36863)
I20260929 16:35:31.252646 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:51467
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:31.278475 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:31.607635 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:31.607920 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:31.608035 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:31.615424 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:31.615542 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:31.615684 22142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:31.625416 22142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:31.626737 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:31.628067 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:31.640826 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:31.640859 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:31.640883 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:31.641538 22142 server_base.cc:1034] running on GCE node
I20260929 16:35:31.642134 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:31.643050 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:31.644243 22142 hybrid_clock.cc:648] HybridClock initialized: now 1790699731644194 us; error 70 us; skew 500 ppm
I20260929 16:35:31.644546 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:31.647503 22142 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:31.648967 22142 webserver.cc:460] Webserver started at http://127.17.205.3:46303/ using document root <none> and password file <none>
I20260929 16:35:31.649637 22142 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:31.649746 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:31.650051 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:31.653004 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "af86f9570a234d2f9a447ff344de11d2"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "fdb4e7f4a3eba8ef9ad021a3f416a8c5"
server_key_iv: "501035e3ca7d04fa75b1f04247c3cc47"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.653749 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "af86f9570a234d2f9a447ff344de11d2"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "fdb4e7f4a3eba8ef9ad021a3f416a8c5"
server_key_iv: "501035e3ca7d04fa75b1f04247c3cc47"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.658864 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:35:31.662336 22157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.663739 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:35:31.663914 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "af86f9570a234d2f9a447ff344de11d2"
format_stamp: "Formatted at 2026-09-29 16:35:31 on dist-test-slave-3h01"
server_key: "fdb4e7f4a3eba8ef9ad021a3f416a8c5"
server_key_iv: "501035e3ca7d04fa75b1f04247c3cc47"
server_key_version: "encryptionkey@0"
I20260929 16:35:31.664098 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:31.694047 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:31.698262 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:31.698532 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:31.699538 22142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:31.700893 22142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:31.700968 22142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.701083 22142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:31.701136 22142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:31.729858 22142 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:39255
I20260929 16:35:31.729895 22269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:39255 every 8 connection(s)
I20260929 16:35:31.731134 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:31.735924 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22142
I20260929 16:35:31.736074 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:31.736485 18228 external_mini_cluster.cc:1428] Setting key d79ecdde89c182c5b0fa0b89de3c82ef
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:31.749877 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.733603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:40685 (local address 127.17.205.62:40785)
0929 16:35:31.733770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.733774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.734947 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 16:35:31.736837 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.736843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.736931 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.737017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.738537 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.739351 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.740921 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.741360 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.741467 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.744689 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.744694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.744697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.744730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.746915 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.747639 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.747645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.747649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.747763 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.748173 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.748178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.748182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.748513 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.748638 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.749328 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.749541 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:35:31.751009 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.734080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40785 (local address 127.17.205.3:40685)
0929 16:35:31.734772 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:31.734827 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:31.736566 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:31.737278 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:31.737286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:31.737783 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:31.738367 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:31.738385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.739493 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.739497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:31.740756 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:31.740772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.741573 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.741580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:31.741816 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.742388 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:31.742411 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:31.744482 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:31.747044 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:31.747053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:31.747072 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:31.747532 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:31.747865 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:31.747870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:31.747874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:31.748052 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:31.748626 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:31.748633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:31.748966 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 16:35:31.749193 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:35:31.749713 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
I20260929 16:35:31.752399 22270 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40785
I20260929 16:35:31.752707 22270 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:31.753522 22270 heartbeater.cc:511] Master 127.17.205.62:40785 requested a full tablet report, sending...
I20260929 16:35:31.754940 21814 ts_manager.cc:194] Registered new tserver with Master: af86f9570a234d2f9a447ff344de11d2 (127.17.205.3:39255)
I20260929 16:35:31.755753 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:40685
I20260929 16:35:31.764074 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:31.783149 22003 heartbeater.cc:503] Master 127.17.205.62:40785 was elected leader, sending a full tablet report...
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:31.810578 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.796659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.17.205.1:41647)
0929 16:35:31.797338 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.797345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.797372 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:35:31.797434 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.797440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.797662 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.797794 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.799179 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.800239 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.801475 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.802046 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.802139 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.805732 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.805744 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.805759 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.805791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.808295 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.808871 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.808877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.808879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.808994 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.809288 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.809290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.809292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.809616 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.809841 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.810034 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.810221 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":348,"threads_started":1}
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:31.826754 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.814683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.17.205.62:40785)
0929 16:35:31.814876 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.814880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.814996 (+ 116us) server_negotiation.cc:407] Connection header received
0929 16:35:31.815121 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.815125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.815185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.815332 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.816431 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.817138 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.818266 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.818692 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.818799 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.822303 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.822308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.822312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.822346 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.824730 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.825233 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.825237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.825240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.825350 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.825607 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.825609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.825611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.825926 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.826096 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.826300 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.826451 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:35:31.827349 21814 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58358
I20260929 16:35:31.828262 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21875
I20260929 16:35:31.839733 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.829030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.17.205.1:41647)
0929 16:35:31.829214 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.829220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.829439 (+ 219us) server_negotiation.cc:407] Connection header received
0929 16:35:31.829595 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.829598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.829657 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.829733 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.830833 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.831528 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.832719 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.833127 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.833240 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.835289 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.835293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.835297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.835329 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.837451 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.838016 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.838022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.838025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.838143 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.838474 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.838480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.838483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.838719 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.838940 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.839237 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.839446 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:35:31.840281 21998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:31.920689 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21875
I20260929 16:35:31.933307 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22009
Sep 29 16:35:31 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:31.947702 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:31.934124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.17.205.2:36863)
0929 16:35:31.934614 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:31.934622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:31.934647 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:35:31.934706 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:31.934712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:31.934938 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:31.935065 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:31.936193 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.937177 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.938321 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:31.938869 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:31.938963 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:31.942810 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:31.942822 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:31.942837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:31.942870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:31.945491 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.946052 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.946058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.946060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.946176 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:31.946448 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:31.946451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:31.946453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:31.946763 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:31.946988 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:31.947196 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 16:35:31.947423 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260929 16:35:31.948922 22132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:32.029088 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22009
I20260929 16:35:32.041605 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22142
Sep 29 16:35:32 dist-test-slave-3h01 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:32.058404 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:32.042581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.17.205.3:39255)
0929 16:35:32.043196 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:32.043204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:32.043263 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:35:32.043330 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:32.043337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:32.043551 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:32.043710 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:32.045044 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:32.046114 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:32.047603 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:32.048208 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:32.048346 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:32.052639 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:32.052650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:32.052665 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:32.052700 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:32.055540 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:32.056231 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:32.056239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:32.056242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:32.056379 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:32.056740 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:32.056746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:32.056748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:32.057151 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:32.057478 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:32.057781 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:35:32.058045 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":310,"threads_started":1}
I20260929 16:35:32.059697 22265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:32.141023 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22142
I20260929 16:35:32.157480 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 21782
I20260929 16:35:32.158625 21844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:32.259378 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 21782
2026-09-29T16:35:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3732 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:32 dist-test-slave-3h01 krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:32 dist-test-slave-3h01 krb5kdc[22289](info): set up 2 sockets
Sep 29 16:35:32 dist-test-slave-3h01 krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260929 16:35:33.259348 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:33 dist-test-slave-3h01 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:33.447013 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40215
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:40215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:33.762404 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:33.762694 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:33.762825 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:33.770057 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:33.770190 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:33.770272 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:33.770350 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:33.770412 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:33.780318 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:40215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:33.781633 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:33.782948 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:33.794037 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:33.794070 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:33.794106 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:33.794461 22302 server_base.cc:1034] running on GCE node
I20260929 16:35:33.795441 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:33.796805 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:33.798027 22302 hybrid_clock.cc:648] HybridClock initialized: now 1790699733797973 us; error 45 us; skew 500 ppm
I20260929 16:35:33.798383 22302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:33 dist-test-slave-3h01 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:33.801402 22302 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:33.802587 22302 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:33.802691 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:33.802996 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:33.806034 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "99c740ebc3a3410e88d8e96b4368886c"
format_stamp: "Formatted at 2026-09-29 16:35:33 on dist-test-slave-3h01"
server_key: "215324978737f59b33c2b5f5b97f52e3"
server_key_iv: "6d19d988a71fd32dd65c70b0276d935c"
server_key_version: "encryptionkey@0"
I20260929 16:35:33.806813 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "99c740ebc3a3410e88d8e96b4368886c"
format_stamp: "Formatted at 2026-09-29 16:35:33 on dist-test-slave-3h01"
server_key: "215324978737f59b33c2b5f5b97f52e3"
server_key_iv: "6d19d988a71fd32dd65c70b0276d935c"
server_key_version: "encryptionkey@0"
I20260929 16:35:33.811895 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:35:33.815657 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:33.816999 22302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:33.817165 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "99c740ebc3a3410e88d8e96b4368886c"
format_stamp: "Formatted at 2026-09-29 16:35:33 on dist-test-slave-3h01"
server_key: "215324978737f59b33c2b5f5b97f52e3"
server_key_iv: "6d19d988a71fd32dd65c70b0276d935c"
server_key_version: "encryptionkey@0"
I20260929 16:35:33.817374 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:33.840834 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:33.845319 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:33.845532 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:33.861405 22302 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:40215
I20260929 16:35:33.861384 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:40215 every 8 connection(s)
I20260929 16:35:33.862298 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:33.863384 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22302
I20260929 16:35:33.863587 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:33.864164 18228 external_mini_cluster.cc:1428] Setting key 0b790ebdad1ddfb119e89fdf935578c9
I20260929 16:35:33.866575 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:35:33 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:33.879443 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: Bootstrap starting.
I20260929 16:35:33.883531 22368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:33.884795 22368 log.cc:824] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:33.888151 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: No bootstrap required, opened a new log
I20260929 16:35:33.888486 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:33.866031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.17.205.62:40215)
0929 16:35:33.866806 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:33.866822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:33.866879 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:35:33.869136 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:33.869166 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:33.869719 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:33.870200 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:33.871601 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:33.873599 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:33.874900 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:33.875932 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:33.876122 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:33.880156 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:33.880168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:33.880188 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:33.880235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:33.883995 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:33.884695 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:33.884704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:33.884713 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:33.884930 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:33.885346 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:33.885352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:33.885356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:33.885986 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:33.886257 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:33.886667 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:35:33.887103 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":318,"threads_started":1}
I20260929 16:35:33.896132 22368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } }
I20260929 16:35:33.896484 22368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:33.896559 22368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c740ebc3a3410e88d8e96b4368886c, State: Initialized, Role: FOLLOWER
I20260929 16:35:33.897113 22368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } }
I20260929 16:35:33.897324 22368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:33.897486 22368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:33.897676 22368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:33.900005 22368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } }
I20260929 16:35:33.900548 22368 leader_election.cc:302] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99c740ebc3a3410e88d8e96b4368886c; no voters:
I20260929 16:35:33.901217 22368 leader_election.cc:288] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:33.901333 22373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:33.902017 22373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [term 1 LEADER]: Becoming Leader. State: Replica: 99c740ebc3a3410e88d8e96b4368886c, State: Running, Role: LEADER
I20260929 16:35:33.902577 22373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } }
I20260929 16:35:33.903422 22368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:33.905447 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99c740ebc3a3410e88d8e96b4368886c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } } }
I20260929 16:35:33.905792 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:33.906208 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c740ebc3a3410e88d8e96b4368886c. Latest consensus state: current_term: 1 leader_uuid: "99c740ebc3a3410e88d8e96b4368886c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c740ebc3a3410e88d8e96b4368886c" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 40215 } } }
I20260929 16:35:33.906518 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:33.907531 22381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:33.912140 22381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:33.920300 22381 catalog_manager.cc:1347] Generated new cluster ID: c65d7d3f7d314e1a9b0146fb3199647a
I20260929 16:35:33.920430 22381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:33.968335 22381 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:33.969641 22381 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:33.975705 22381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: Generated new TSK 0
I20260929 16:35:33.976536 22381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:34.027158 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:34.345218 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:34.345504 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:34.345628 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:34.353171 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:34.353286 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:34.353434 22394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:34.363234 22394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:34.364573 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:34.365908 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:34.378341 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:34.378393 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:34.378341 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:34.379516 22394 server_base.cc:1034] running on GCE node
I20260929 16:35:34.380137 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:34.381098 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:34.382293 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790699734382232 us; error 57 us; skew 500 ppm
I20260929 16:35:34.382596 22394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:34 dist-test-slave-3h01 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699734, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:34.385524 22394 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:34.386682 22394 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:34.386778 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:34.387034 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:34.390065 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "83f27db41c80478f92bbd49e1fd10d5b"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "38f24e3512735d2a4eba8d3ded761259"
server_key_iv: "ae43f4202d5e4fc99c141e670074078c"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.390853 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "83f27db41c80478f92bbd49e1fd10d5b"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "38f24e3512735d2a4eba8d3ded761259"
server_key_iv: "ae43f4202d5e4fc99c141e670074078c"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.396003 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:35:34.399561 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.400945 22394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:34.401090 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "83f27db41c80478f92bbd49e1fd10d5b"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "38f24e3512735d2a4eba8d3ded761259"
server_key_iv: "ae43f4202d5e4fc99c141e670074078c"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.401300 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:34.423553 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:34.427408 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:34.427647 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:34.428594 22394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:34.429893 22394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:34.429976 22394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.430080 22394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:34.430151 22394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.459157 22394 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42899
I20260929 16:35:34.459201 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42899 every 8 connection(s)
I20260929 16:35:34.460013 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:34.464637 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22394
I20260929 16:35:34.464856 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:34.465628 18228 external_mini_cluster.cc:1428] Setting key 12d8641f385977006490a717c75c3873
Sep 29 16:35:34 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699734, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:34.481600 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:34.462413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:50679 (local address 127.17.205.62:40215)
0929 16:35:34.462845 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:34.462852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:34.464021 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 16:35:34.466519 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:34.466524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:34.466613 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:34.466705 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:34.468637 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.469321 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.471443 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.471842 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.471933 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:34.475945 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:34.475952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:34.475956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:34.475994 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:34.478327 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:34.479204 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:34.479245 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:34.479249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:34.479392 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:34.479833 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:34.479836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:34.479838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:34.480108 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:34.480217 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:34.481124 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 16:35:34.481307 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":192,"threads_started":1}
I20260929 16:35:34.483155 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:34.463042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40215 (local address 127.17.205.1:50679)
0929 16:35:34.463842 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:34.463894 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:34.466210 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:34.466974 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:34.466983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:34.467606 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:34.468463 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:34.468489 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.469487 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.469492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:34.471323 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:34.471338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.472050 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.472056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:34.472318 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:34.473020 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:34.473047 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:34.475656 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:34.478461 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:34.478470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:34.478489 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:34.479021 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:34.479496 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:34.479501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:34.479505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:34.479731 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:34.480225 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:34.480233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:34.480714 (+ 481us) client_negotiation.cc:769] Sending connection context
0929 16:35:34.481021 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 16:35:34.481686 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:35:34.484869 22521 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40215
I20260929 16:35:34.485162 22521 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:34.485795 22521 heartbeater.cc:511] Master 127.17.205.62:40215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:35:34.487924 22333 ts_manager.cc:194] Registered new tserver with Master: 83f27db41c80478f92bbd49e1fd10d5b (127.17.205.1:42899)
I20260929 16:35:34.489852 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:50679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:34.510165 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:34.823403 22527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:34.823698 22527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:34.823832 22527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:34.831008 22527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:34.831151 22527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:34.831326 22527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:34.841303 22527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:34.842614 22527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:34.844023 22527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:34.856918 22532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:34.856973 22533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:34.856892 22535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:34.858223 22527 server_base.cc:1034] running on GCE node
I20260929 16:35:34.858842 22527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:34.859867 22527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:34.861065 22527 hybrid_clock.cc:648] HybridClock initialized: now 1790699734861009 us; error 43 us; skew 500 ppm
I20260929 16:35:34.861402 22527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:34 dist-test-slave-3h01 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699734, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:34.864595 22527 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:34.865862 22527 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:34.865965 22527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:34.866262 22527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:34.869189 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "631ad11afad241c9bcd20a2cbd918eec"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "4bec9d57b15d312d65b5f02b6ea6307b"
server_key_iv: "97e15fde6b2a1a070a8a348e75ea7ac9"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.869995 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "631ad11afad241c9bcd20a2cbd918eec"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "4bec9d57b15d312d65b5f02b6ea6307b"
server_key_iv: "97e15fde6b2a1a070a8a348e75ea7ac9"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.875348 22527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 16:35:34.878937 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.880429 22527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:34.880601 22527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "631ad11afad241c9bcd20a2cbd918eec"
format_stamp: "Formatted at 2026-09-29 16:35:34 on dist-test-slave-3h01"
server_key: "4bec9d57b15d312d65b5f02b6ea6307b"
server_key_iv: "97e15fde6b2a1a070a8a348e75ea7ac9"
server_key_version: "encryptionkey@0"
I20260929 16:35:34.880820 22527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:34.896176 22527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:34.900601 22527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:34.900872 22527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:34.901862 22527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:34.903282 22527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:34.903381 22527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.903486 22527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:34.903553 22527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:34.932430 22527 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:34293
I20260929 16:35:34.932474 22653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:34293 every 8 connection(s)
I20260929 16:35:34.933266 22527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:34.937654 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22527
I20260929 16:35:34.937829 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:34.938232 18228 external_mini_cluster.cc:1428] Setting key 61c6b77d9b771b074f9fda01448c1a51
Sep 29 16:35:34 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699734, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:34.953835 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:34.935643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:43677 (local address 127.17.205.62:40215)
0929 16:35:34.935796 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:34.935800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:34.937003 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 16:35:34.938948 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:34.938954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:34.939060 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:34.939159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:34.941421 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.942416 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.944015 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.944545 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.944661 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:34.948241 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:34.948246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:34.948250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:34.948283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:34.950640 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:34.951546 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:34.951550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:34.951552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:34.951682 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:34.952109 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:34.952112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:34.952114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:34.952412 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:34.952519 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:34.953405 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 16:35:34.953571 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:35:34.955487 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:34.936140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40215 (local address 127.17.205.2:43677)
0929 16:35:34.936844 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:34.936890 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:34.938608 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:34.940064 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:34.940076 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:34.940645 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:34.941238 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:34.941256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.942545 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.942548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:34.943877 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:34.943889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:34.944731 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:34.944736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:34.944927 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:34.945566 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:34.945594 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:34.947990 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:34.950775 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:34.950784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:34.950804 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:34.951442 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:34.951776 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:34.951780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:34.951784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:34.952016 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:34.952537 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:34.952545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:34.953022 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 16:35:34.953357 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 16:35:34.953940 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:35:34.957036 22654 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40215
I20260929 16:35:34.957314 22654 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:34.958017 22654 heartbeater.cc:511] Master 127.17.205.62:40215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:35:34.959600 22333 ts_manager.cc:194] Registered new tserver with Master: 631ad11afad241c9bcd20a2cbd918eec (127.17.205.2:34293)
I20260929 16:35:34.960717 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:43677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:34.981823 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:35.302649 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:35.302958 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:35.303089 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:35.310573 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:35.310706 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:35.310868 22659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:35.320948 22659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:40215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:35.322333 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:35.323690 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:35.336535 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:35.336620 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:35.336644 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:35.337271 22659 server_base.cc:1034] running on GCE node
I20260929 16:35:35.337894 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:35.338941 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:35.340159 22659 hybrid_clock.cc:648] HybridClock initialized: now 1790699735340085 us; error 69 us; skew 500 ppm
I20260929 16:35:35.340509 22659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:35 dist-test-slave-3h01 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699735, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:35.348026 22659 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:35.349349 22659 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:35.349468 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:35.349774 22659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:35.352998 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "973216cc62f2490a84b50ed94e29ca2f"
format_stamp: "Formatted at 2026-09-29 16:35:35 on dist-test-slave-3h01"
server_key: "32a5e2bc9c69d86ee6b2c300aef22d2b"
server_key_iv: "2b9d733d4d05c706c47815d100c22a9c"
server_key_version: "encryptionkey@0"
I20260929 16:35:35.353785 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "973216cc62f2490a84b50ed94e29ca2f"
format_stamp: "Formatted at 2026-09-29 16:35:35 on dist-test-slave-3h01"
server_key: "32a5e2bc9c69d86ee6b2c300aef22d2b"
server_key_iv: "2b9d733d4d05c706c47815d100c22a9c"
server_key_version: "encryptionkey@0"
I20260929 16:35:35.359026 22659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:35:35.362649 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:35.364384 22659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:35.364574 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "973216cc62f2490a84b50ed94e29ca2f"
format_stamp: "Formatted at 2026-09-29 16:35:35 on dist-test-slave-3h01"
server_key: "32a5e2bc9c69d86ee6b2c300aef22d2b"
server_key_iv: "2b9d733d4d05c706c47815d100c22a9c"
server_key_version: "encryptionkey@0"
I20260929 16:35:35.364799 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:35.375768 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:35.380031 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:35.380285 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:35.381244 22659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:35.382601 22659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:35.382704 22659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:35.382809 22659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:35.382879 22659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:35.413311 22659 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43767
I20260929 16:35:35.413367 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43767 every 8 connection(s)
I20260929 16:35:35.414143 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:35.420064 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22659
I20260929 16:35:35.420292 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:35.420732 18228 external_mini_cluster.cc:1428] Setting key 188fc896b643f244cc98e92a84d80701
Sep 29 16:35:35 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699735, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:35.433876 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.416548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:34715 (local address 127.17.205.62:40215)
0929 16:35:35.416739 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.416746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.417935 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 16:35:35.419893 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.419899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.419994 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.420109 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:35.422008 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.422922 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.424707 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.425122 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.425239 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.428552 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:35.428558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:35.428561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:35.428597 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:35.431084 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.431859 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.431863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.431866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.431979 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.432382 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.432386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.432388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.432664 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:35.432788 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.433450 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.433617 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:35:35.435163 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.417056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:40215 (local address 127.17.205.3:34715)
0929 16:35:35.417768 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.417812 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.419602 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.420372 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.420386 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.421007 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:35.421804 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.421829 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.423070 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.423075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.424560 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.424572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.425307 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.425311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.425506 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.426110 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:35.426132 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:35.428321 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:35.431283 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:35.431291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:35.431307 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:35.431744 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:35.432064 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:35.432067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:35.432070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:35.432249 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:35.432766 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:35.432771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:35.433138 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.433374 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.433785 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260929 16:35:35.436743 22786 heartbeater.cc:347] Connected to a master server at 127.17.205.62:40215
I20260929 16:35:35.437023 22786 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:35.437633 22786 heartbeater.cc:511] Master 127.17.205.62:40215 requested a full tablet report, sending...
I20260929 16:35:35.439050 22333 ts_manager.cc:194] Registered new tserver with Master: 973216cc62f2490a84b50ed94e29ca2f (127.17.205.3:43767)
I20260929 16:35:35.439869 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:34715
I20260929 16:35:35.448709 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:35.466413 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.453315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.17.205.62:40215)
0929 16:35:35.453500 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.453504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.453936 (+ 432us) server_negotiation.cc:407] Connection header received
0929 16:35:35.454064 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.454068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.454143 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.454252 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:35.455661 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.456460 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.457798 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.458291 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.458403 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.460732 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:35.460747 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:35.460753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:35.460797 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:35.463985 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.464728 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.464732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.464735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.464876 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.465164 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.465167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.465169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.465462 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:35.465640 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.465992 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.466168 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:35:35.471040 22333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:35.478022 22333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:35.493443 22521 heartbeater.cc:503] Master 127.17.205.62:40215 was elected leader, sending a full tablet report...
I20260929 16:35:35.502359 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.490677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.17.205.1:42899)
0929 16:35:35.491321 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.491331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.492023 (+ 692us) server_negotiation.cc:407] Connection header received
0929 16:35:35.492491 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.492502 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.492804 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.492978 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:35.493986 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.495923 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.499013 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.500602 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.500752 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.501019 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.501772 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.501971 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":310,"threads_started":1}
I20260929 16:35:35.503417 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.490381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42899 (local address 127.0.0.1:48536)
0929 16:35:35.491873 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.491916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.492199 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.493317 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.493325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.493366 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:35.493790 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.493803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.496133 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.496140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.498757 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.498782 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.501176 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.501183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.501456 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.501513 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.502812 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.503021 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"thread_start_us":8685,"threads_started":1}
I20260929 16:35:35.509150 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.490522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43767 (local address 127.0.0.1:47566)
0929 16:35:35.499409 (+ 8887us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.499436 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.499573 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.500725 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.500731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.500761 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:35.501218 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.501230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.503449 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.503455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.505828 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.505846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.508235 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.508243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.508454 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.508494 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.508589 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.508752 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8723,"thread_start_us":972,"threads_started":1}
I20260929 16:35:35.509934 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.491021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.17.205.3:43767)
0929 16:35:35.492685 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.492694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.499719 (+ 7025us) server_negotiation.cc:407] Connection header received
0929 16:35:35.499850 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.499861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.500188 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.500431 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:35.501489 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.503257 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.506042 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.507692 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.507822 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.508079 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.509464 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.509633 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1315,"threads_started":1}
I20260929 16:35:35.508949 22456 tablet_service.cc:1467] Processing CreateTablet for tablet 88973a762615429c92803e2fe25ba3e5 (DEFAULT_TABLE table=test-table [id=da0dea37325e43709383d78fff8f08df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:35.511231 22456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88973a762615429c92803e2fe25ba3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:35.513928 22721 tablet_service.cc:1467] Processing CreateTablet for tablet 88973a762615429c92803e2fe25ba3e5 (DEFAULT_TABLE table=test-table [id=da0dea37325e43709383d78fff8f08df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:35.515182 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.492200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.17.205.2:34293)
0929 16:35:35.508344 (+ 16144us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.508353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.508383 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:35:35.508476 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.508484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.508763 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.508962 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:35.510125 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.511494 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.512925 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.513958 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.514082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.514281 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.514746 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.514912 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15970,"thread_start_us":16930,"threads_started":1}
I20260929 16:35:35.515476 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88973a762615429c92803e2fe25ba3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:35.515692 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.491990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:34293 (local address 127.0.0.1:51604)
0929 16:35:35.503881 (+ 11891us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.503903 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.504011 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.509746 (+ 5735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.509752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.509780 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:35.509950 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.509961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.511630 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.511633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.512765 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.512777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.514248 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.514254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.514441 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.514472 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.515266 (+ 794us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.515440 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11752,"thread_start_us":17458,"threads_started":1}
I20260929 16:35:35.518813 22589 tablet_service.cc:1467] Processing CreateTablet for tablet 88973a762615429c92803e2fe25ba3e5 (DEFAULT_TABLE table=test-table [id=da0dea37325e43709383d78fff8f08df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:35.520339 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88973a762615429c92803e2fe25ba3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:35.526957 22804 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Bootstrap starting.
I20260929 16:35:35.529834 22805 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Bootstrap starting.
I20260929 16:35:35.531387 22804 tablet_bootstrap.cc:654] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:35.532675 22804 log.cc:824] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:35.532868 22806 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Bootstrap starting.
I20260929 16:35:35.533735 22805 tablet_bootstrap.cc:654] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:35.534963 22805 log.cc:824] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:35.535794 22804 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: No bootstrap required, opened a new log
I20260929 16:35:35.536068 22804 ts_tablet_manager.cc:1397] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:35:35.536144 22806 tablet_bootstrap.cc:654] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:35.537009 22806 log.cc:824] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:35.537943 22805 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: No bootstrap required, opened a new log
I20260929 16:35:35.538221 22805 ts_tablet_manager.cc:1397] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:35:35.539333 22806 tablet_bootstrap.cc:492] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: No bootstrap required, opened a new log
I20260929 16:35:35.539565 22806 ts_tablet_manager.cc:1397] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:35:35.546487 22806 raft_consensus.cc:348] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.546459 22804 raft_consensus.cc:348] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.546864 22806 raft_consensus.cc:374] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:35.546962 22806 raft_consensus.cc:729] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631ad11afad241c9bcd20a2cbd918eec, State: Initialized, Role: FOLLOWER
I20260929 16:35:35.546999 22804 raft_consensus.cc:374] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:35.547093 22804 raft_consensus.cc:729] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f27db41c80478f92bbd49e1fd10d5b, State: Initialized, Role: FOLLOWER
I20260929 16:35:35.547688 22806 consensus_queue.cc:260] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.547726 22804 consensus_queue.cc:260] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.548394 22805 raft_consensus.cc:348] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.548768 22654 heartbeater.cc:503] Master 127.17.205.62:40215 was elected leader, sending a full tablet report...
I20260929 16:35:35.548825 22805 raft_consensus.cc:374] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:35.548915 22805 raft_consensus.cc:729] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973216cc62f2490a84b50ed94e29ca2f, State: Initialized, Role: FOLLOWER
I20260929 16:35:35.549450 22806 ts_tablet_manager.cc:1428] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260929 16:35:35.549935 22805 consensus_queue.cc:260] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.551646 22805 ts_tablet_manager.cc:1428] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
I20260929 16:35:35.551680 22804 ts_tablet_manager.cc:1428] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 16:35:35.552196 22786 heartbeater.cc:503] Master 127.17.205.62:40215 was elected leader, sending a full tablet report...
I20260929 16:35:35.555737 22810 raft_consensus.cc:482] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:35.555979 22810 raft_consensus.cc:504] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.557508 22810 leader_election.cc:288] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f27db41c80478f92bbd49e1fd10d5b (127.17.205.1:42899), 973216cc62f2490a84b50ed94e29ca2f (127.17.205.3:43767)
I20260929 16:35:35.564020 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.557979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57247 (local address 127.17.205.3:43767)
0929 16:35:35.558152 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.558158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.558911 (+ 753us) server_negotiation.cc:407] Connection header received
0929 16:35:35.559029 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.559034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.559097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.559175 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:35.559767 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.560532 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.562426 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.563144 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.563327 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.563407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.563652 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.563775 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:35:35.564028 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.557964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:36699 (local address 127.17.205.1:42899)
0929 16:35:35.558143 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.558149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.558954 (+ 805us) server_negotiation.cc:407] Connection header received
0929 16:35:35.559173 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.559179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.559294 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.559384 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:35.559920 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.560629 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.562367 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.563097 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.563250 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.563325 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.563656 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.563786 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:35:35.564307 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.558089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42899 (local address 127.17.205.2:36699)
0929 16:35:35.558856 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.558871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.559043 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.559401 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.559404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.559424 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:35.559784 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.559794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.560746 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.560749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.562222 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.562233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.563328 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.563335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.563517 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.563540 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.563615 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.563778 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":76,"thread_start_us":398,"threads_started":1}
I20260929 16:35:35.564349 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.558070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43767 (local address 127.17.205.2:57247)
0929 16:35:35.558811 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:35.558826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:35.558937 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:35.559198 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:35.559202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:35.559253 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:35.559637 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.559647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.560648 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.560653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.562281 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:35.562294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.563376 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.563381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:35.563522 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.563543 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:35:35.563619 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:35:35.563791 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":112,"thread_start_us":419,"threads_started":1}
I20260929 16:35:35.565191 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88973a762615429c92803e2fe25ba3e5" candidate_uuid: "631ad11afad241c9bcd20a2cbd918eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f27db41c80478f92bbd49e1fd10d5b" is_pre_election: true
I20260929 16:35:35.565200 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88973a762615429c92803e2fe25ba3e5" candidate_uuid: "631ad11afad241c9bcd20a2cbd918eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973216cc62f2490a84b50ed94e29ca2f" is_pre_election: true
I20260929 16:35:35.565708 22741 raft_consensus.cc:2454] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631ad11afad241c9bcd20a2cbd918eec in term 0.
I20260929 16:35:35.565729 22476 raft_consensus.cc:2454] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631ad11afad241c9bcd20a2cbd918eec in term 0.
I20260929 16:35:35.566367 22542 leader_election.cc:302] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631ad11afad241c9bcd20a2cbd918eec, 973216cc62f2490a84b50ed94e29ca2f; no voters:
I20260929 16:35:35.566754 22810 raft_consensus.cc:2781] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:35.566895 22810 raft_consensus.cc:482] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:35.566994 22810 raft_consensus.cc:3037] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:35.569190 22810 raft_consensus.cc:504] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.569780 22810 leader_election.cc:288] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [CANDIDATE]: Term 1 election: Requested vote from peers 83f27db41c80478f92bbd49e1fd10d5b (127.17.205.1:42899), 973216cc62f2490a84b50ed94e29ca2f (127.17.205.3:43767)
I20260929 16:35:35.570173 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88973a762615429c92803e2fe25ba3e5" candidate_uuid: "631ad11afad241c9bcd20a2cbd918eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f27db41c80478f92bbd49e1fd10d5b"
I20260929 16:35:35.570256 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88973a762615429c92803e2fe25ba3e5" candidate_uuid: "631ad11afad241c9bcd20a2cbd918eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973216cc62f2490a84b50ed94e29ca2f"
I20260929 16:35:35.570389 22476 raft_consensus.cc:3037] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:35.570451 22741 raft_consensus.cc:3037] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:35.573480 22476 raft_consensus.cc:2454] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631ad11afad241c9bcd20a2cbd918eec in term 1.
I20260929 16:35:35.573480 22741 raft_consensus.cc:2454] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631ad11afad241c9bcd20a2cbd918eec in term 1.
I20260929 16:35:35.573889 22545 leader_election.cc:302] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631ad11afad241c9bcd20a2cbd918eec, 83f27db41c80478f92bbd49e1fd10d5b; no voters:
I20260929 16:35:35.574154 22810 raft_consensus.cc:2781] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:35.574561 22810 raft_consensus.cc:686] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 1 LEADER]: Becoming Leader. State: Replica: 631ad11afad241c9bcd20a2cbd918eec, State: Running, Role: LEADER
I20260929 16:35:35.575119 22810 consensus_queue.cc:237] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } }
I20260929 16:35:35.579517 22333 catalog_manager.cc:5495] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec reported cstate change: term changed from 0 to 1, leader changed from <none> to 631ad11afad241c9bcd20a2cbd918eec (127.17.205.2). New cstate: current_term: 1 leader_uuid: "631ad11afad241c9bcd20a2cbd918eec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631ad11afad241c9bcd20a2cbd918eec" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260929 16:35:35.613494 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.608213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.17.205.2:34293)
0929 16:35:35.608408 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.608414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.608537 (+ 123us) server_negotiation.cc:407] Connection header received
0929 16:35:35.608796 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.608800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.608879 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.609012 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:35.609385 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.610086 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.611409 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.611823 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.611932 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.612416 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:35.612935 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:35.613071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.613145 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.613227 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:35.621773 22741 raft_consensus.cc:1261] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 1 FOLLOWER]: Refusing update from remote peer 631ad11afad241c9bcd20a2cbd918eec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:35.621798 22476 raft_consensus.cc:1261] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 1 FOLLOWER]: Refusing update from remote peer 631ad11afad241c9bcd20a2cbd918eec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:35.622659 22815 consensus_queue.cc:1035] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "973216cc62f2490a84b50ed94e29ca2f" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:35.622995 22810 consensus_queue.cc:1035] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f27db41c80478f92bbd49e1fd10d5b" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:35.633976 22818 mvcc.cc:205] Tried to move back new op lower bound from 7334706117096980480 to 7334706116919726080. Current Snapshot: MvccSnapshot[applied={T|T < 7334706117096980480}]
I20260929 16:35:35.635820 22820 mvcc.cc:205] Tried to move back new op lower bound from 7334706117096980480 to 7334706116919726080. Current Snapshot: MvccSnapshot[applied={T|T < 7334706117096980480}]
I20260929 16:35:35.648448 22331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:35.648794 22331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:35.652735 22331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99c740ebc3a3410e88d8e96b4368886c: Sending DeleteTablet for 3 replicas of tablet 88973a762615429c92803e2fe25ba3e5
I20260929 16:35:35.653888 22721 tablet_service.cc:1514] Processing DeleteTablet for tablet 88973a762615429c92803e2fe25ba3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:35 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:47566
I20260929 16:35:35.653890 22589 tablet_service.cc:1514] Processing DeleteTablet for tablet 88973a762615429c92803e2fe25ba3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:35 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:51604
I20260929 16:35:35.653903 22456 tablet_service.cc:1514] Processing DeleteTablet for tablet 88973a762615429c92803e2fe25ba3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:35 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:48536
I20260929 16:35:35.657703 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22394
I20260929 16:35:35.670639 22829 tablet_replica.cc:330] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: stopping tablet replica
I20260929 16:35:35.671137 22828 tablet_replica.cc:330] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: stopping tablet replica
I20260929 16:35:35.671657 22828 raft_consensus.cc:2229] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:35.671695 22829 raft_consensus.cc:2229] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:35.672420 22828 raft_consensus.cc:2258] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:35.672875 22829 raft_consensus.cc:2258] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:35.673242 22827 tablet_replica.cc:330] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: stopping tablet replica
I20260929 16:35:35.673662 22827 raft_consensus.cc:2229] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:35.674144 22827 raft_consensus.cc:2258] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:35.675072 22828 ts_tablet_manager.cc:1905] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:35.677053 22829 ts_tablet_manager.cc:1905] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:35.682308 22828 ts_tablet_manager.cc:1918] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:35.682498 22828 log.cc:1196] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/88973a762615429c92803e2fe25ba3e5
I20260929 16:35:35.683045 22828 ts_tablet_manager.cc:1939] T 88973a762615429c92803e2fe25ba3e5 P 631ad11afad241c9bcd20a2cbd918eec: Deleting consensus metadata
I20260929 16:35:35.684155 22829 ts_tablet_manager.cc:1918] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:35.684337 22829 log.cc:1196] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/88973a762615429c92803e2fe25ba3e5
I20260929 16:35:35.684957 22827 ts_tablet_manager.cc:1905] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:35.685029 22319 catalog_manager.cc:4841] TS 631ad11afad241c9bcd20a2cbd918eec (127.17.205.2:34293): tablet 88973a762615429c92803e2fe25ba3e5 (table test-table [id=da0dea37325e43709383d78fff8f08df]) successfully deleted
I20260929 16:35:35.685429 22829 ts_tablet_manager.cc:1939] T 88973a762615429c92803e2fe25ba3e5 P 83f27db41c80478f92bbd49e1fd10d5b: Deleting consensus metadata
I20260929 16:35:35.686851 22320 catalog_manager.cc:4841] TS 83f27db41c80478f92bbd49e1fd10d5b (127.17.205.1:42899): tablet 88973a762615429c92803e2fe25ba3e5 (table test-table [id=da0dea37325e43709383d78fff8f08df]) successfully deleted
Sep 29 16:35:35 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:35.691833 22827 ts_tablet_manager.cc:1918] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:35.692008 22827 log.cc:1196] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/88973a762615429c92803e2fe25ba3e5
I20260929 16:35:35.692530 22827 ts_tablet_manager.cc:1939] T 88973a762615429c92803e2fe25ba3e5 P 973216cc62f2490a84b50ed94e29ca2f: Deleting consensus metadata
I20260929 16:35:35.694392 22317 catalog_manager.cc:4841] TS 973216cc62f2490a84b50ed94e29ca2f (127.17.205.3:43767): tablet 88973a762615429c92803e2fe25ba3e5 (table test-table [id=da0dea37325e43709383d78fff8f08df]) successfully deleted
I20260929 16:35:35.695300 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.658938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.17.205.1:42899)
0929 16:35:35.659102 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.659108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.674605 (+ 15497us) server_negotiation.cc:407] Connection header received
0929 16:35:35.674686 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.674691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.674783 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.674906 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:35.676760 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.677894 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.679990 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.680663 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.680761 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.688933 (+ 8172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:35.688946 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:35.688960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:35.689000 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:35.692238 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.692968 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.692976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.692978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.693115 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.693482 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.693486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.693488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.693861 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:35.694141 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.694659 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.694895 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:35:35.695927 22516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:35.820986 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22394
I20260929 16:35:35.834921 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22527
Sep 29 16:35:35 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:35.850840 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.835753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.17.205.2:34293)
0929 16:35:35.835946 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.835952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.835978 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:35:35.836091 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.836097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.836188 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.836305 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:35.837406 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.838606 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.840058 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.840766 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.840901 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.845103 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:35.845119 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:35.845136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:35.845179 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:35.848117 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.848932 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.848939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.848942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.849083 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:35.849461 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:35.849464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:35.849466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:35.849813 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:35.850054 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:35.850345 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:35:35.850517 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:35:35.851605 22649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:35.973183 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22527
I20260929 16:35:35.988003 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22659
Sep 29 16:35:35 dist-test-slave-3h01 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:36.002080 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:35.988756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.17.205.3:43767)
0929 16:35:35.988958 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:35.988964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:35.988984 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:35:35.989118 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:35.989124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:35.989194 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:35.989278 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:35.990527 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.991301 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.992566 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:35.992988 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:35.993089 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:35.997153 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:35.997169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:35.997186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:35.997221 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:35.999791 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:36.000378 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:36.000384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:36.000387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:36.000522 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:36.000805 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:36.000808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:36.000810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:36.001159 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:36.001354 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:36.001588 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:35:36.001763 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:35:36.002707 22781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:36.088562 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22659
I20260929 16:35:36.100999 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22302
I20260929 16:35:36.102053 22363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:36.205229 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22302
2026-09-29T16:35:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3945 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:36Z Disabled control of system clock
I20260929 16:35:36.265201 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45251
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:35:36.580955 22838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:36.581261 22838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:36.581343 22838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:36.588656 22838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:36.588788 22838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:36.588824 22838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:36.588879 22838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:36.588920 22838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:36.598213 22838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:45251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:36.599535 22838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:36.601429 22838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:36.612964 22843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:36.612895 22844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:36.612876 22846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:36.614048 22838 server_base.cc:1034] running on GCE node
I20260929 16:35:36.614897 22838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:36.616423 22838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:36.617630 22838 hybrid_clock.cc:648] HybridClock initialized: now 1790699736617569 us; error 56 us; skew 500 ppm
I20260929 16:35:36.617974 22838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:36.620200 22838 webserver.cc:460] Webserver started at http://127.17.205.62:43883/ using document root <none> and password file <none>
I20260929 16:35:36.620941 22838 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:36.621039 22838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:36.621353 22838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:36.624286 22838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "e8f7ae5b55e444c1bb551fd16b3669e9"
format_stamp: "Formatted at 2026-09-29 16:35:36 on dist-test-slave-3h01"
server_key: "b8178c73d8af3c160798741c7d6c60bf"
server_key_iv: "080ac69aabf144666417eb284d9876c1"
server_key_version: "encryptionkey@0"
I20260929 16:35:36.625038 22838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "e8f7ae5b55e444c1bb551fd16b3669e9"
format_stamp: "Formatted at 2026-09-29 16:35:36 on dist-test-slave-3h01"
server_key: "b8178c73d8af3c160798741c7d6c60bf"
server_key_iv: "080ac69aabf144666417eb284d9876c1"
server_key_version: "encryptionkey@0"
I20260929 16:35:36.629985 22838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:35:36.633312 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:36.634624 22838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:36.634779 22838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "e8f7ae5b55e444c1bb551fd16b3669e9"
format_stamp: "Formatted at 2026-09-29 16:35:36 on dist-test-slave-3h01"
server_key: "b8178c73d8af3c160798741c7d6c60bf"
server_key_iv: "080ac69aabf144666417eb284d9876c1"
server_key_version: "encryptionkey@0"
I20260929 16:35:36.634968 22838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:36.640597 22838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:36.641324 22838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:36.641579 22838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:36.657959 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:45251 every 8 connection(s)
I20260929 16:35:36.657956 22838 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:45251
I20260929 16:35:36.659363 22838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:36.662482 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22838
I20260929 16:35:36.662668 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:36.663517 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:36.663506 18228 external_mini_cluster.cc:1428] Setting key 923da659f285163c2db25e3657464a95
I20260929 16:35:36.671737 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:36.665596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.17.205.62:45251)
0929 16:35:36.666564 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:36.666574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:36.666619 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:35:36.668201 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:36.668225 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:36.668663 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:36.669012 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:36.670072 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:36.670079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:36.670093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:36.670106 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:36.670169 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:36.670189 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:36.670371 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:36.670477 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:35:36.670743 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":330,"threads_started":1}
I20260929 16:35:36.676306 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: Bootstrap starting.
I20260929 16:35:36.679186 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:36.680197 22904 log.cc:824] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:36.682893 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: No bootstrap required, opened a new log
I20260929 16:35:36.689886 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } }
I20260929 16:35:36.690263 22904 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:36.690366 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f7ae5b55e444c1bb551fd16b3669e9, State: Initialized, Role: FOLLOWER
I20260929 16:35:36.690923 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } }
I20260929 16:35:36.691155 22904 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:36.691295 22904 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:36.691452 22904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:36.693522 22904 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } }
I20260929 16:35:36.694023 22904 leader_election.cc:302] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f7ae5b55e444c1bb551fd16b3669e9; no voters:
I20260929 16:35:36.694602 22904 leader_election.cc:288] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:36.694718 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:36.695399 22909 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [term 1 LEADER]: Becoming Leader. State: Replica: e8f7ae5b55e444c1bb551fd16b3669e9, State: Running, Role: LEADER
I20260929 16:35:36.695911 22909 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } }
I20260929 16:35:36.696655 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:36.698470 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } } }
I20260929 16:35:36.698865 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:36.699342 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f7ae5b55e444c1bb551fd16b3669e9. Latest consensus state: current_term: 1 leader_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f7ae5b55e444c1bb551fd16b3669e9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45251 } } }
I20260929 16:35:36.699648 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:36.701122 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:36.706387 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:36.715940 22914 catalog_manager.cc:1347] Generated new cluster ID: d6fbc8af3f1a47f198dd673d915f2b5c
I20260929 16:35:36.716065 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:36.775045 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:36.775496 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:36.786846 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: Generated new TSK 0
I20260929 16:35:36.787793 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:36.803615 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--builtin_ntp_servers=127.17.205.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:35:37.132503 22928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:37.132838 22928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:37.132969 22928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:37.140399 22928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:37.140533 22928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:37.140677 22928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:37.150017 22928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:37.151371 22928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:37.153195 22928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:37.168287 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:37.168263 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:37.168390 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:37.169294 22928 server_base.cc:1034] running on GCE node
I20260929 16:35:37.169905 22928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:37.170966 22928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:37.172186 22928 hybrid_clock.cc:648] HybridClock initialized: now 1790699737172137 us; error 44 us; skew 500 ppm
I20260929 16:35:37.172498 22928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:37.174650 22928 webserver.cc:460] Webserver started at http://127.17.205.1:41419/ using document root <none> and password file <none>
I20260929 16:35:37.175395 22928 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:37.175495 22928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:37.175812 22928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:37.178606 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "be4a0a8433144e0db89b13c6b2349056"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "aeea98b1493070f20bfbd28025676943"
server_key_iv: "f35bce3eb762bc525239c6c57b2c0fde"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.179436 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "be4a0a8433144e0db89b13c6b2349056"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "aeea98b1493070f20bfbd28025676943"
server_key_iv: "f35bce3eb762bc525239c6c57b2c0fde"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.184625 22928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:37.188366 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.189906 22928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:37.190080 22928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "be4a0a8433144e0db89b13c6b2349056"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "aeea98b1493070f20bfbd28025676943"
server_key_iv: "f35bce3eb762bc525239c6c57b2c0fde"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.190302 22928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:37.196115 22928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:37.197000 22928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:37.197261 22928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:37.198213 22928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:37.199640 22928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:37.199733 22928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.199852 22928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:37.199947 22928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.229950 22928 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:38967
I20260929 16:35:37.230091 23054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:38967 every 8 connection(s)
I20260929 16:35:37.231335 22928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:37.232110 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 22928
I20260929 16:35:37.232302 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:37.233055 18228 external_mini_cluster.cc:1428] Setting key 84c0b29b631a5ad821d1f8aa0f4d4369
I20260929 16:35:37.235991 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--builtin_ntp_servers=127.17.205.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:35:37.240760 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:37.233989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:49773 (local address 127.17.205.62:45251)
0929 16:35:37.234445 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:37.234451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:37.236067 (+ 1616us) server_negotiation.cc:407] Connection header received
0929 16:35:37.238364 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:37.238370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:37.238435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:37.238525 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:37.240013 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:37.240018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:37.240021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:37.240038 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:37.240078 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:37.240090 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:37.240188 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:37.240454 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:35:37.240514 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260929 16:35:37.241961 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:37.234734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45251 (local address 127.17.205.1:49773)
0929 16:35:37.235892 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:37.235947 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:37.238049 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:37.238779 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:37.238789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:37.238934 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:37.239659 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:37.239662 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:37.239717 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:37.239846 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:37.240173 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:37.240182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:37.240189 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:35:37.240489 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 16:35:37.240762 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":281,"threads_started":1}
I20260929 16:35:37.243877 23055 heartbeater.cc:347] Connected to a master server at 127.17.205.62:45251
I20260929 16:35:37.244197 23055 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:37.244908 23055 heartbeater.cc:511] Master 127.17.205.62:45251 requested a full tablet report, sending...
I20260929 16:35:37.247308 22869 ts_manager.cc:194] Registered new tserver with Master: be4a0a8433144e0db89b13c6b2349056 (127.17.205.1:38967)
W20260929 16:35:37.566955 23059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:37.567305 23059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:37.567421 23059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:37.574841 23059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:37.574968 23059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:37.575088 23059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:37.584430 23059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:37.585726 23059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:37.587620 23059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:37.601123 23064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:37.601209 23065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:37.601154 23067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:37.604871 23059 server_base.cc:1034] running on GCE node
I20260929 16:35:37.605505 23059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:37.606499 23059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:37.607733 23059 hybrid_clock.cc:648] HybridClock initialized: now 1790699737607684 us; error 52 us; skew 500 ppm
I20260929 16:35:37.608100 23059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:37.610566 23059 webserver.cc:460] Webserver started at http://127.17.205.2:35541/ using document root <none> and password file <none>
I20260929 16:35:37.611362 23059 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:37.611474 23059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:37.611797 23059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:37.614722 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "411da8a9f69b48e2881043463e879706"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "ed0ed11ec6d13351b785e3158797b27c"
server_key_iv: "3b62b177865815746395252de311359c"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.615551 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "411da8a9f69b48e2881043463e879706"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "ed0ed11ec6d13351b785e3158797b27c"
server_key_iv: "3b62b177865815746395252de311359c"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.620739 23059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:35:37.624325 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.625787 23059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:37.625963 23059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "411da8a9f69b48e2881043463e879706"
format_stamp: "Formatted at 2026-09-29 16:35:37 on dist-test-slave-3h01"
server_key: "ed0ed11ec6d13351b785e3158797b27c"
server_key_iv: "3b62b177865815746395252de311359c"
server_key_version: "encryptionkey@0"
I20260929 16:35:37.626178 23059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:37.631961 23059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:37.632766 23059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:37.633009 23059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:37.633903 23059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:37.635289 23059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:37.635381 23059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.635488 23059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:37.635555 23059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:37.672843 23059 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:44331
I20260929 16:35:37.672880 23185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:44331 every 8 connection(s)
I20260929 16:35:37.674263 23059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:37.678725 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23059
I20260929 16:35:37.678895 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:37.679458 18228 external_mini_cluster.cc:1428] Setting key c724fb34ecfb197b9dafc93fadbd9856
I20260929 16:35:37.682451 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--builtin_ntp_servers=127.17.205.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:35:37.684365 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:37.677536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:35853 (local address 127.17.205.62:45251)
0929 16:35:37.677739 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:37.677745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:37.679611 (+ 1866us) server_negotiation.cc:407] Connection header received
0929 16:35:37.681953 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:37.681957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:37.682021 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:37.682106 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:37.683739 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:37.683744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:37.683747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:37.683763 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:37.683794 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:37.683806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:37.683899 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:37.684106 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:35:37.684156 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:35:37.685663 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:37.678124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45251 (local address 127.17.205.2:35853)
0929 16:35:37.679397 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:37.679453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:37.681608 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:37.682405 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:37.682418 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:37.682565 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:37.683335 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:37.683339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:37.683396 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:37.683515 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:37.683888 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:37.683897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:37.683904 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:35:37.684183 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 16:35:37.684471 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":681,"threads_started":1}
I20260929 16:35:37.687067 23186 heartbeater.cc:347] Connected to a master server at 127.17.205.62:45251
I20260929 16:35:37.687439 23186 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:37.688158 23186 heartbeater.cc:511] Master 127.17.205.62:45251 requested a full tablet report, sending...
I20260929 16:35:37.689671 22869 ts_manager.cc:194] Registered new tserver with Master: 411da8a9f69b48e2881043463e879706 (127.17.205.2:44331)
W20260929 16:35:38.005436 23189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:38.005731 23189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:38.005817 23189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:38.013125 23189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:38.013247 23189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:38.013372 23189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:38.022612 23189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:38.023990 23189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:38.025763 23189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:38.039076 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:38.039062 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:38.039109 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:38.039793 23189 server_base.cc:1034] running on GCE node
I20260929 16:35:38.040570 23189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:38.041643 23189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:38.042869 23189 hybrid_clock.cc:648] HybridClock initialized: now 1790699738042815 us; error 48 us; skew 500 ppm
I20260929 16:35:38.043236 23189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:38.045471 23189 webserver.cc:460] Webserver started at http://127.17.205.3:38989/ using document root <none> and password file <none>
I20260929 16:35:38.046160 23189 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:38.046265 23189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:38.046567 23189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:38.049486 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61"
format_stamp: "Formatted at 2026-09-29 16:35:38 on dist-test-slave-3h01"
server_key: "29391255b90bad85cfe16353c8973e03"
server_key_iv: "13547744d763a00b539f816a4dcab6da"
server_key_version: "encryptionkey@0"
I20260929 16:35:38.050268 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61"
format_stamp: "Formatted at 2026-09-29 16:35:38 on dist-test-slave-3h01"
server_key: "29391255b90bad85cfe16353c8973e03"
server_key_iv: "13547744d763a00b539f816a4dcab6da"
server_key_version: "encryptionkey@0"
I20260929 16:35:38.055740 23189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:35:38.059466 23203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:38.061074 23189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:38.061256 23189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61"
format_stamp: "Formatted at 2026-09-29 16:35:38 on dist-test-slave-3h01"
server_key: "29391255b90bad85cfe16353c8973e03"
server_key_iv: "13547744d763a00b539f816a4dcab6da"
server_key_version: "encryptionkey@0"
I20260929 16:35:38.061494 23189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:38.067301 23189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:38.068099 23189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:38.068351 23189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:38.069235 23189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:38.070613 23189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:38.070705 23189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:38.070808 23189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:38.070874 23189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:38.100664 23189 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43917
I20260929 16:35:38.100754 23315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43917 every 8 connection(s)
I20260929 16:35:38.102026 23189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:38.102757 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23189
I20260929 16:35:38.102918 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:38.103518 18228 external_mini_cluster.cc:1428] Setting key 0313387f932187afe5cb4979e2bd1429
I20260929 16:35:38.110532 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.104691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:33199 (local address 127.17.205.62:45251)
0929 16:35:38.104851 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.104855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.106575 (+ 1720us) server_negotiation.cc:407] Connection header received
0929 16:35:38.108283 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.108288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.108365 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.108454 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.109821 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.109827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.109832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.109852 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.109884 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.109896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.109996 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.110181 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.110245 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:35:38.111564 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.105306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45251 (local address 127.17.205.3:33199)
0929 16:35:38.106417 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.106464 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.108027 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.108745 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.108754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.108860 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.109510 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.109513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.109553 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.109644 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.109983 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.109992 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.110000 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.110262 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.110550 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":287,"threads_started":1}
I20260929 16:35:38.112906 23316 heartbeater.cc:347] Connected to a master server at 127.17.205.62:45251
I20260929 16:35:38.113209 23316 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:38.113807 23316 heartbeater.cc:511] Master 127.17.205.62:45251 requested a full tablet report, sending...
I20260929 16:35:38.115182 22869 ts_manager.cc:194] Registered new tserver with Master: f4d2c2c516d64ef49251b7ecf7f2ed61 (127.17.205.3:43917)
I20260929 16:35:38.118643 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:35:38.125391 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.122970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.17.205.62:45251)
0929 16:35:38.123149 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.123156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.123657 (+ 501us) server_negotiation.cc:407] Connection header received
0929 16:35:38.123829 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.123833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.123887 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.123965 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.124791 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.124797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.124801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.124819 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.124854 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.124865 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.124972 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.125068 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.125125 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:35:38.129073 22869 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:38.135740 22869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:38.163754 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.147979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.17.205.3:43917)
0929 16:35:38.158221 (+ 10242us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.158232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.160431 (+ 2199us) server_negotiation.cc:407] Connection header received
0929 16:35:38.160687 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.160696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.160933 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.161053 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.162898 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.162904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.162915 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.162930 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.162978 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.162995 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.163160 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.163260 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.163365 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10017,"thread_start_us":10261,"threads_started":1}
I20260929 16:35:38.163748 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.147678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.17.205.2:44331)
0929 16:35:38.159752 (+ 12074us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.159761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.159792 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:38.159884 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.159892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.160183 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.160338 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.162825 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.162836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.162852 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.162879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.162928 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.162944 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.163070 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.163296 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.163368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11910,"thread_start_us":11790,"threads_started":1}
I20260929 16:35:38.163784 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.147578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:44331 (local address 127.0.0.1:58194)
0929 16:35:38.159059 (+ 11481us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.159118 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.159274 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.161574 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.161583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.161691 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.162522 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.162524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.162552 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.162629 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.163039 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.163050 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.163059 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.163314 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.163402 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11191,"thread_start_us":12474,"threads_started":1}
I20260929 16:35:38.166335 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.147754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43917 (local address 127.0.0.1:42094)
0929 16:35:38.160323 (+ 12569us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.160340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.160536 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.162109 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.162114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.162136 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.162721 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.162724 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.162738 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.162773 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.163103 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.163107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.163112 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.165762 (+ 2650us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.165829 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12426,"thread_start_us":10860,"threads_started":1}
I20260929 16:35:38.166744 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.160386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59766)
0929 16:35:38.163281 (+ 2895us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.163297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.163380 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.164481 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.164486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.164509 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.165516 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.165520 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.165535 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.165575 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.166018 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.166027 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.166033 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.166189 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.166244 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2751,"thread_start_us":1864,"threads_started":1}
I20260929 16:35:38.166712 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.160009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.17.205.1:38967)
0929 16:35:38.162763 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.162771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.163491 (+ 720us) server_negotiation.cc:407] Connection header received
0929 16:35:38.163565 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.163572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.163810 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.163954 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.165767 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.165776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.165791 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.165817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.165881 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.165904 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.166036 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.166215 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.166325 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2595,"thread_start_us":2598,"threads_started":1}
I20260929 16:35:38.168021 23121 tablet_service.cc:1467] Processing CreateTablet for tablet 1687e4690d6b47b693a9f79852b4224c (DEFAULT_TABLE table=test-table [id=9bb6cf8028f8448b8fd1b0875d3a2d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:38.169533 23121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1687e4690d6b47b693a9f79852b4224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:38.169816 23251 tablet_service.cc:1467] Processing CreateTablet for tablet 1687e4690d6b47b693a9f79852b4224c (DEFAULT_TABLE table=test-table [id=9bb6cf8028f8448b8fd1b0875d3a2d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:38.171092 23251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1687e4690d6b47b693a9f79852b4224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:38.170667 22990 tablet_service.cc:1467] Processing CreateTablet for tablet 1687e4690d6b47b693a9f79852b4224c (DEFAULT_TABLE table=test-table [id=9bb6cf8028f8448b8fd1b0875d3a2d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:38.171965 22990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1687e4690d6b47b693a9f79852b4224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:38.182327 23334 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Bootstrap starting.
I20260929 16:35:38.185720 23334 tablet_bootstrap.cc:654] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:38.186486 23336 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Bootstrap starting.
I20260929 16:35:38.188148 23335 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Bootstrap starting.
I20260929 16:35:38.189078 23334 log.cc:824] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:38.190213 23336 tablet_bootstrap.cc:654] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:38.191072 23335 tablet_bootstrap.cc:654] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:38.191509 23336 log.cc:824] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:38.192059 23335 log.cc:824] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:38.192478 23334 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: No bootstrap required, opened a new log
I20260929 16:35:38.192836 23334 ts_tablet_manager.cc:1397] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:35:38.194588 23336 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: No bootstrap required, opened a new log
I20260929 16:35:38.194893 23336 ts_tablet_manager.cc:1397] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 16:35:38.194983 23335 tablet_bootstrap.cc:492] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: No bootstrap required, opened a new log
I20260929 16:35:38.195320 23335 ts_tablet_manager.cc:1397] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:35:38.200963 23334 raft_consensus.cc:348] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.201397 23334 raft_consensus.cc:374] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:38.201535 23334 raft_consensus.cc:729] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d2c2c516d64ef49251b7ecf7f2ed61, State: Initialized, Role: FOLLOWER
I20260929 16:35:38.202162 23334 consensus_queue.cc:260] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.203281 23316 heartbeater.cc:503] Master 127.17.205.62:45251 was elected leader, sending a full tablet report...
I20260929 16:35:38.204013 23334 ts_tablet_manager.cc:1428] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:35:38.205267 23335 raft_consensus.cc:348] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.205237 23336 raft_consensus.cc:348] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.205686 23335 raft_consensus.cc:374] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:38.205686 23336 raft_consensus.cc:374] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:38.205785 23335 raft_consensus.cc:729] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411da8a9f69b48e2881043463e879706, State: Initialized, Role: FOLLOWER
I20260929 16:35:38.205780 23336 raft_consensus.cc:729] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4a0a8433144e0db89b13c6b2349056, State: Initialized, Role: FOLLOWER
I20260929 16:35:38.206409 23335 consensus_queue.cc:260] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.206408 23336 consensus_queue.cc:260] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.207677 23186 heartbeater.cc:503] Master 127.17.205.62:45251 was elected leader, sending a full tablet report...
I20260929 16:35:38.208271 23335 ts_tablet_manager.cc:1428] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:35:38.210614 23336 ts_tablet_manager.cc:1428] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260929 16:35:38.211081 23055 heartbeater.cc:503] Master 127.17.205.62:45251 was elected leader, sending a full tablet report...
W20260929 16:35:38.233700 23056 tablet.cc:2369] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:35:38.236730 23342 raft_consensus.cc:482] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:38.237059 23342 raft_consensus.cc:504] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.238967 23342 leader_election.cc:288] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 411da8a9f69b48e2881043463e879706 (127.17.205.2:44331), f4d2c2c516d64ef49251b7ecf7f2ed61 (127.17.205.3:43917)
I20260929 16:35:38.242640 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.239556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:47511 (local address 127.17.205.3:43917)
0929 16:35:38.239728 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.239735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.240772 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 16:35:38.240886 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.240891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.240953 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.241069 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.242073 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.242077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.242081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.242096 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.242126 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.242137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.242240 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.242329 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.242394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:35:38.242632 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.239418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:39335 (local address 127.17.205.2:44331)
0929 16:35:38.239599 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.239606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.240332 (+ 726us) server_negotiation.cc:407] Connection header received
0929 16:35:38.240629 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.240635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.240700 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.240770 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.241976 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.241983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.241988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.242007 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.242042 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.242055 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.242218 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.242282 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.242352 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:35:38.242890 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.239359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:44331 (local address 127.17.205.1:39335)
0929 16:35:38.240193 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.240217 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.240456 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.240948 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.240951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.240970 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.241725 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.241729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.241748 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.241798 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.242163 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.242169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.242175 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.242315 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.242373 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":514,"threads_started":1}
I20260929 16:35:38.243274 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.239976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43917 (local address 127.17.205.1:47511)
0929 16:35:38.240662 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.240678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:38.240770 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:38.241099 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:38.241103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:38.241125 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:38.241889 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:35:38.241892 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:38.241907 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:35:38.241944 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:38.242219 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:38.242224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:38.242228 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:35:38.242360 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 16:35:38.242406 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":372,"threads_started":1}
I20260929 16:35:38.243856 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1687e4690d6b47b693a9f79852b4224c" candidate_uuid: "be4a0a8433144e0db89b13c6b2349056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411da8a9f69b48e2881043463e879706" is_pre_election: true
I20260929 16:35:38.244307 23141 raft_consensus.cc:2454] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4a0a8433144e0db89b13c6b2349056 in term 0.
I20260929 16:35:38.244270 23271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1687e4690d6b47b693a9f79852b4224c" candidate_uuid: "be4a0a8433144e0db89b13c6b2349056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" is_pre_election: true
I20260929 16:35:38.244740 23271 raft_consensus.cc:2454] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4a0a8433144e0db89b13c6b2349056 in term 0.
I20260929 16:35:38.244926 22946 leader_election.cc:302] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411da8a9f69b48e2881043463e879706, be4a0a8433144e0db89b13c6b2349056; no voters:
I20260929 16:35:38.245286 23342 raft_consensus.cc:2781] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:38.245414 23342 raft_consensus.cc:482] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:38.245512 23342 raft_consensus.cc:3037] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:38.248023 23342 raft_consensus.cc:504] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.248754 23342 leader_election.cc:288] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [CANDIDATE]: Term 1 election: Requested vote from peers 411da8a9f69b48e2881043463e879706 (127.17.205.2:44331), f4d2c2c516d64ef49251b7ecf7f2ed61 (127.17.205.3:43917)
I20260929 16:35:38.249240 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1687e4690d6b47b693a9f79852b4224c" candidate_uuid: "be4a0a8433144e0db89b13c6b2349056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411da8a9f69b48e2881043463e879706"
I20260929 16:35:38.249460 23141 raft_consensus.cc:3037] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:38.249673 23271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1687e4690d6b47b693a9f79852b4224c" candidate_uuid: "be4a0a8433144e0db89b13c6b2349056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61"
I20260929 16:35:38.249890 23271 raft_consensus.cc:3037] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:38.251854 23141 raft_consensus.cc:2454] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4a0a8433144e0db89b13c6b2349056 in term 1.
I20260929 16:35:38.252203 22946 leader_election.cc:302] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411da8a9f69b48e2881043463e879706, be4a0a8433144e0db89b13c6b2349056; no voters:
I20260929 16:35:38.252380 23271 raft_consensus.cc:2454] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4a0a8433144e0db89b13c6b2349056 in term 1.
I20260929 16:35:38.252487 23342 raft_consensus.cc:2781] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:38.252992 23342 raft_consensus.cc:686] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 1 LEADER]: Becoming Leader. State: Replica: be4a0a8433144e0db89b13c6b2349056, State: Running, Role: LEADER
I20260929 16:35:38.253561 23342 consensus_queue.cc:237] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } }
I20260929 16:35:38.258167 22868 catalog_manager.cc:5495] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 reported cstate change: term changed from 0 to 1, leader changed from <none> to be4a0a8433144e0db89b13c6b2349056 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "be4a0a8433144e0db89b13c6b2349056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be4a0a8433144e0db89b13c6b2349056" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260929 16:35:38.281411 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.279537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.17.205.1:38967)
0929 16:35:38.279722 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.279727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.279744 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:35:38.279931 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.279935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.280015 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.280097 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.280866 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.280870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.280873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.280889 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.280920 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.280932 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.281026 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.281082 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.281126 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:35:38.289553 23271 raft_consensus.cc:1261] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 1 FOLLOWER]: Refusing update from remote peer be4a0a8433144e0db89b13c6b2349056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:38.289554 23141 raft_consensus.cc:1261] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 1 FOLLOWER]: Refusing update from remote peer be4a0a8433144e0db89b13c6b2349056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:38.290496 23345 consensus_queue.cc:1035] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "411da8a9f69b48e2881043463e879706" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:38.290844 23342 consensus_queue.cc:1035] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d2c2c516d64ef49251b7ecf7f2ed61" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:38.319941 22869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:38.320326 22869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:38.324276 22869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8f7ae5b55e444c1bb551fd16b3669e9: Sending DeleteTablet for 3 replicas of tablet 1687e4690d6b47b693a9f79852b4224c
I20260929 16:35:38.325548 23121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1687e4690d6b47b693a9f79852b4224c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:38 UTC) from {username='slave'} at 127.0.0.1:58194
I20260929 16:35:38.325551 23251 tablet_service.cc:1514] Processing DeleteTablet for tablet 1687e4690d6b47b693a9f79852b4224c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:38 UTC) from {username='slave'} at 127.0.0.1:42094
I20260929 16:35:38.325744 22990 tablet_service.cc:1514] Processing DeleteTablet for tablet 1687e4690d6b47b693a9f79852b4224c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:38 UTC) from {username='slave'} at 127.0.0.1:59766
I20260929 16:35:38.327747 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22928
I20260929 16:35:38.338649 23360 tablet_replica.cc:330] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: stopping tablet replica
I20260929 16:35:38.339496 23359 tablet_replica.cc:330] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: stopping tablet replica
I20260929 16:35:38.339967 23359 raft_consensus.cc:2229] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:38.340530 23359 raft_consensus.cc:2258] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:38.339180 23360 raft_consensus.cc:2229] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:38.341733 23360 raft_consensus.cc:2258] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:38.344080 23359 ts_tablet_manager.cc:1905] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:38.349308 23360 ts_tablet_manager.cc:1905] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:38.351370 23359 ts_tablet_manager.cc:1918] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:38.351655 23359 log.cc:1196] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/1687e4690d6b47b693a9f79852b4224c
I20260929 16:35:38.352226 23359 ts_tablet_manager.cc:1939] T 1687e4690d6b47b693a9f79852b4224c P 411da8a9f69b48e2881043463e879706: Deleting consensus metadata
I20260929 16:35:38.354939 22856 catalog_manager.cc:4841] TS 411da8a9f69b48e2881043463e879706 (127.17.205.2:44331): tablet 1687e4690d6b47b693a9f79852b4224c (table test-table [id=9bb6cf8028f8448b8fd1b0875d3a2d64]) successfully deleted
I20260929 16:35:38.356539 23358 tablet_replica.cc:330] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: stopping tablet replica
I20260929 16:35:38.356880 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.329546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.17.205.1:38967)
0929 16:35:38.329747 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.329754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.352863 (+ 23109us) server_negotiation.cc:407] Connection header received
0929 16:35:38.353406 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.353413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.353497 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.353576 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.356093 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.356099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.356104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.356126 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.356164 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.356179 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.356287 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.356529 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.356601 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:35:38.357295 23360 ts_tablet_manager.cc:1918] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:38.357393 23358 raft_consensus.cc:2229] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:38.357518 23360 log.cc:1196] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/1687e4690d6b47b693a9f79852b4224c
I20260929 16:35:38.357963 23358 raft_consensus.cc:2258] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:38.357911 23050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:38.360353 23358 ts_tablet_manager.cc:1905] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:38.366802 23358 ts_tablet_manager.cc:1918] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:38.366951 23358 log.cc:1196] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/1687e4690d6b47b693a9f79852b4224c
I20260929 16:35:38.367390 23358 ts_tablet_manager.cc:1939] T 1687e4690d6b47b693a9f79852b4224c P f4d2c2c516d64ef49251b7ecf7f2ed61: Deleting consensus metadata
I20260929 16:35:38.368904 22853 catalog_manager.cc:4841] TS f4d2c2c516d64ef49251b7ecf7f2ed61 (127.17.205.3:43917): tablet 1687e4690d6b47b693a9f79852b4224c (table test-table [id=9bb6cf8028f8448b8fd1b0875d3a2d64]) successfully deleted
I20260929 16:35:38.455626 23360 ts_tablet_manager.cc:1939] T 1687e4690d6b47b693a9f79852b4224c P be4a0a8433144e0db89b13c6b2349056: Deleting consensus metadata
I20260929 16:35:38.455837 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22928
I20260929 16:35:38.470685 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23059
W20260929 16:35:38.470736 22853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:38967 (error 108)
I20260929 16:35:38.472445 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.472045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59784)
0929 16:35:38.472197 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.472313 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:35:38.472695 22853 catalog_manager.cc:4584] TS be4a0a8433144e0db89b13c6b2349056 (127.17.205.1:38967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1687e4690d6b47b693a9f79852b4224c: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
I20260929 16:35:38.473567 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.471424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.17.205.2:44331)
0929 16:35:38.471577 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.471581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.471830 (+ 249us) server_negotiation.cc:407] Connection header received
0929 16:35:38.472074 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.472079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.472163 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.472244 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.473095 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.473099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.473103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.473117 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.473147 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.473158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.473253 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.473321 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.473372 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:35:38.474146 23181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:38.526257 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.525705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59790)
0929 16:35:38.525934 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.526057 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:35:38.527281 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.526874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59796)
0929 16:35:38.527012 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.527092 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 16:35:38.559811 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23059
I20260929 16:35:38.572796 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23189
I20260929 16:35:38.575541 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.573484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.17.205.3:43917)
0929 16:35:38.573653 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:38.573658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:38.573682 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:35:38.573872 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:38.573877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:38.573951 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:38.574032 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:38.574887 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:38.574893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:38.574899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:38.574916 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:38.574948 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:38.574961 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:38.575072 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:38.575175 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:35:38.575277 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:35:38.576093 23311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:38.587936 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.587469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59804)
0929 16:35:38.587676 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.587788 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:35:38.588848 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.588550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59806)
0929 16:35:38.588669 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.588734 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 16:35:38.664609 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23189
I20260929 16:35:38.669435 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.668926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59810)
0929 16:35:38.669154 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.669261 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:35:38.670607 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:38.670152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38967 (local address 127.0.0.1:59818)
0929 16:35:38.670379 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:38.670462 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:38967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:35:38.678279 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 22838
I20260929 16:35:38.679412 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:38.782896 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 22838
2026-09-29T16:35:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2577 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:35:38.819684 18228 mini_oidc.cc:152] Starting JWKS server
I20260929 16:35:38.822202 18228 webserver.cc:460] Webserver started at https://127.0.0.1:43171/ using document root <none> and password file <none>
I20260929 16:35:38.822468 18228 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:35:38.823094 18228 webserver.cc:460] Webserver started at http://127.0.0.1:42173/ using document root <none> and password file <none>
2026-09-29T16:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:38Z Disabled control of system clock
I20260929 16:35:38.848316 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33963
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:33963
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/testchainca.cert with env {}
W20260929 16:35:39.167202 23372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:39.167523 23372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:39.167632 23372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:39.174736 23372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:39.174852 23372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:39.174911 23372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:39.174985 23372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:39.175035 23372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:35:39.175114 23372 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:35:39.175158 23372 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43171/jwks.json
I20260929 16:35:39.185073 23372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:33963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:39.186368 23372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:39.187692 23372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:39.199098 23380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:39.199342 23377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:39.199131 23378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:39.199991 23372 server_base.cc:1034] running on GCE node
I20260929 16:35:39.200764 23372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:39.202232 23372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:39.203420 23372 hybrid_clock.cc:648] HybridClock initialized: now 1790699739203375 us; error 37 us; skew 500 ppm
I20260929 16:35:39.203763 23372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:39.206103 23372 webserver.cc:460] Webserver started at http://127.17.205.62:36735/ using document root <none> and password file <none>
I20260929 16:35:39.206838 23372 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:39.206934 23372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:39.207484 23372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:39.210605 23372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "cf25b196e55247f5a5df94eedebad3ec"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "c81c4f125e7dede5aab8b7cafeb28d3f"
server_key_iv: "2e687c7a78241d9f58ace0abecd5350e"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.211524 23372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "cf25b196e55247f5a5df94eedebad3ec"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "c81c4f125e7dede5aab8b7cafeb28d3f"
server_key_iv: "2e687c7a78241d9f58ace0abecd5350e"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.217209 23372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:39.220731 23386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:39.222141 23372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:35:39.222308 23372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "cf25b196e55247f5a5df94eedebad3ec"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "c81c4f125e7dede5aab8b7cafeb28d3f"
server_key_iv: "2e687c7a78241d9f58ace0abecd5350e"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.222512 23372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:39.259083 23372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:39.259981 23372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:39.260272 23372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:39.276991 23372 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:33963
I20260929 16:35:39.277010 23437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:33963 every 8 connection(s)
I20260929 16:35:39.278332 23372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:39.282299 23438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:39.286820 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23372
I20260929 16:35:39.287029 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:39.287830 18228 external_mini_cluster.cc:1428] Setting key e23665387457c7cf80929de0d498a715
I20260929 16:35:39.293959 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec: Bootstrap starting.
I20260929 16:35:39.297153 23438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:39.298195 23438 log.cc:824] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:39.301031 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec: No bootstrap required, opened a new log
I20260929 16:35:39.301216 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:39.289815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.17.205.62:33963)
0929 16:35:39.290814 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:39.290837 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:39.290883 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:35:39.292671 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:39.292697 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:39.293204 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:39.293583 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:39.294162 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.295528 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.296850 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.297581 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.297758 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:39.298884 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:39.298892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:39.298906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:39.299504 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:39.299584 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:39.299606 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:39.299706 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:39.299940 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:35:39.300239 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":328,"threads_started":1}
I20260929 16:35:39.305037 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:39.309172 23438 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } }
I20260929 16:35:39.309537 23438 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:39.309626 23438 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf25b196e55247f5a5df94eedebad3ec, State: Initialized, Role: FOLLOWER
I20260929 16:35:39.310231 23438 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } }
I20260929 16:35:39.310456 23438 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:39.310578 23438 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:39.310743 23438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:39.312938 23438 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } }
I20260929 16:35:39.313396 23438 leader_election.cc:302] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf25b196e55247f5a5df94eedebad3ec; no voters:
I20260929 16:35:39.313871 23438 leader_election.cc:288] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:39.313994 23443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:39.314635 23443 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [term 1 LEADER]: Becoming Leader. State: Replica: cf25b196e55247f5a5df94eedebad3ec, State: Running, Role: LEADER
I20260929 16:35:39.315129 23443 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } }
I20260929 16:35:39.315815 23438 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:39.317869 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf25b196e55247f5a5df94eedebad3ec. Latest consensus state: current_term: 1 leader_uuid: "cf25b196e55247f5a5df94eedebad3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } } }
I20260929 16:35:39.317931 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf25b196e55247f5a5df94eedebad3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf25b196e55247f5a5df94eedebad3ec" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33963 } } }
I20260929 16:35:39.318279 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:39.318311 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:39.321178 23452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:39.326472 23452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:39.336347 23452 catalog_manager.cc:1347] Generated new cluster ID: 284356400c0944a896ff50407a68af0c
I20260929 16:35:39.336478 23452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:39.369066 23452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:39.370355 23452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:39.378931 23452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf25b196e55247f5a5df94eedebad3ec: Generated new TSK 0
I20260929 16:35:39.379781 23452 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:35:39.419265 23468 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:35:39.419476 23468 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:35:39.420418 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:39.399485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.17.205.62:33963)
0929 16:35:39.399721 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:39.399729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:39.400253 (+ 524us) server_negotiation.cc:407] Connection header received
0929 16:35:39.400610 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:39.400614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:39.400689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:39.400810 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:35:39.401431 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.402160 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.404214 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.404673 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.404774 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:39.405428 (+ 654us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:35:39.419842 (+ 14414us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:35:39.420167 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 16:35:39.420580 23441 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 16:35:39.422408 23467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:39.399351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33963 (local address 127.0.0.1:50160)
0929 16:35:39.400138 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:39.400162 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:39.400362 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:39.400872 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:39.400877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:39.400891 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:35:39.401278 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:39.401290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.402294 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.402300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:39.404060 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:39.404079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:39.404846 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:39.404853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:39.405052 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:39.405140 (+ 88us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:35:39.420155 (+ 15015us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:35:39.420544 (+ 389us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.205.62:33963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":422,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.205.62:33963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:35:39.424118 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23372
I20260929 16:35:39.425251 23433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:39.518867 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23372
2026-09-29T16:35:39Z chronyd exiting
I20260929 16:35:39.566039 18228 test_util.cc:175] -----------------------------------------------
I20260929 16:35:39.566176 18228 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699695913193-18228-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:35:39.569774 18228 mini_oidc.cc:152] Starting JWKS server
I20260929 16:35:39.571557 18228 webserver.cc:460] Webserver started at https://127.0.0.1:42139/ using document root <none> and password file <none>
I20260929 16:35:39.571744 18228 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:35:39.572359 18228 webserver.cc:460] Webserver started at http://127.0.0.1:40437/ using document root <none> and password file <none>
2026-09-29T16:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:39Z Disabled control of system clock
I20260929 16:35:39.596876 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41111
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:41111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/testchainca.cert with env {}
W20260929 16:35:39.908960 23475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:39.909281 23475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:39.909420 23475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:39.916682 23475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:39.916805 23475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:39.916877 23475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:39.916954 23475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:39.917014 23475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:35:39.917095 23475 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:35:39.917148 23475 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42139/jwks.json
I20260929 16:35:39.927277 23475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:39.928603 23475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:39.929958 23475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:39.941416 23481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:39.941378 23480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:39.941490 23483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:39.942131 23475 server_base.cc:1034] running on GCE node
I20260929 16:35:39.943032 23475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:39.944509 23475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:39.945717 23475 hybrid_clock.cc:648] HybridClock initialized: now 1790699739945672 us; error 42 us; skew 500 ppm
I20260929 16:35:39.946089 23475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:39.948377 23475 webserver.cc:460] Webserver started at http://127.17.205.62:34095/ using document root <none> and password file <none>
I20260929 16:35:39.949083 23475 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:39.949186 23475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:39.949488 23475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:39.952543 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "a607f8ef32954be9b6ab7068ecaa210e"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "28c50c916d6eb89873aed4850c16c0df"
server_key_iv: "9ebed632e08ff9294d097c89066d5101"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.953337 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "a607f8ef32954be9b6ab7068ecaa210e"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "28c50c916d6eb89873aed4850c16c0df"
server_key_iv: "9ebed632e08ff9294d097c89066d5101"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.958528 23475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:35:39.962199 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:39.963732 23475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:39.963927 23475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a607f8ef32954be9b6ab7068ecaa210e"
format_stamp: "Formatted at 2026-09-29 16:35:39 on dist-test-slave-3h01"
server_key: "28c50c916d6eb89873aed4850c16c0df"
server_key_iv: "9ebed632e08ff9294d097c89066d5101"
server_key_version: "encryptionkey@0"
I20260929 16:35:39.964131 23475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:39.988896 23475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:39.989769 23475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:39.990077 23475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:40.006842 23475 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:41111
I20260929 16:35:40.006872 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:41111 every 8 connection(s)
I20260929 16:35:40.008285 23475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:40.012097 23541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:40.013087 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23475
I20260929 16:35:40.013280 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:40.013890 18228 external_mini_cluster.cc:1428] Setting key 02ef26bb474492b25984feaf263ceaf5
I20260929 16:35:40.024920 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e: Bootstrap starting.
I20260929 16:35:40.027325 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:40.015891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.17.205.62:41111)
0929 16:35:40.017015 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:40.017027 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:40.017075 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:35:40.018640 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:40.018670 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:40.019264 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:40.019673 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:40.020243 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.021519 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.022836 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.023568 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.023741 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.024865 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:40.024872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:40.024887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:40.025454 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:40.025532 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:40.025553 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:40.025648 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:40.025886 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:35:40.026195 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":386,"threads_started":1}
I20260929 16:35:40.028426 23541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:40.029479 23541 log.cc:824] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:40.031118 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:40.032222 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e: No bootstrap required, opened a new log
I20260929 16:35:40.039139 23541 raft_consensus.cc:348] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } }
I20260929 16:35:40.039568 23541 raft_consensus.cc:374] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:40.039662 23541 raft_consensus.cc:729] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a607f8ef32954be9b6ab7068ecaa210e, State: Initialized, Role: FOLLOWER
I20260929 16:35:40.040220 23541 consensus_queue.cc:260] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } }
I20260929 16:35:40.040427 23541 raft_consensus.cc:388] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:40.040570 23541 raft_consensus.cc:482] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:40.040707 23541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:40.042766 23541 raft_consensus.cc:504] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } }
I20260929 16:35:40.043375 23541 leader_election.cc:302] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a607f8ef32954be9b6ab7068ecaa210e; no voters:
I20260929 16:35:40.043933 23541 leader_election.cc:288] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:40.044070 23546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:40.044744 23546 raft_consensus.cc:686] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [term 1 LEADER]: Becoming Leader. State: Replica: a607f8ef32954be9b6ab7068ecaa210e, State: Running, Role: LEADER
I20260929 16:35:40.045301 23546 consensus_queue.cc:237] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } }
I20260929 16:35:40.045871 23541 sys_catalog.cc:564] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:40.048158 23547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a607f8ef32954be9b6ab7068ecaa210e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } } }
I20260929 16:35:40.048514 23547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:40.048306 23548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a607f8ef32954be9b6ab7068ecaa210e. Latest consensus state: current_term: 1 leader_uuid: "a607f8ef32954be9b6ab7068ecaa210e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a607f8ef32954be9b6ab7068ecaa210e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41111 } } }
I20260929 16:35:40.048659 23548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:40.050009 23552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:40.055428 23552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:40.064925 23552 catalog_manager.cc:1347] Generated new cluster ID: 487b7ef1133741ed95b09ab38308c3a2
I20260929 16:35:40.065061 23552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:40.110029 23552 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:40.111872 23552 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:40.128549 23552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a607f8ef32954be9b6ab7068ecaa210e: Generated new TSK 0
I20260929 16:35:40.129431 23552 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:35:40.166795 23571 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:35:40.166920 23571 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:35:40.168223 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:40.148115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.17.205.62:41111)
0929 16:35:40.148316 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:40.148322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:40.148601 (+ 279us) server_negotiation.cc:407] Connection header received
0929 16:35:40.148848 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:40.148854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:40.148957 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:40.149131 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:35:40.149535 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.150669 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.152180 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.152870 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.152993 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.153463 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:35:40.167522 (+ 14059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:35:40.167928 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:35:40.168349 23544 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:35:40.168483 23570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:40.147943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41111 (local address 127.0.0.1:40792)
0929 16:35:40.148496 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:40.148512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:40.148658 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:40.149143 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:40.149146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:40.149155 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:35:40.149389 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:40.149398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.150826 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.150832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:40.152029 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:40.152045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.153034 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.153039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:40.153167 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.153201 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:35:40.167797 (+ 14596us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:35:40.168024 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.205.62:41111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260929 16:35:40.169984 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23475
I20260929 16:35:40.171082 23536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:40.267880 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23475
2026-09-29T16:35:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:35:40.319852 18228 mini_oidc.cc:152] Starting JWKS server
I20260929 16:35:40.321676 18228 webserver.cc:460] Webserver started at https://127.0.0.1:39113/ using document root <none> and password file <none>
I20260929 16:35:40.321893 18228 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:35:40.322497 18228 webserver.cc:460] Webserver started at http://127.0.0.1:33299/ using document root <none> and password file <none>
2026-09-29T16:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:40Z Disabled control of system clock
I20260929 16:35:40.346628 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44395
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:44395
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39113/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:35:40.656817 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:40.657092 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:40.657174 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:40.664355 23578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:40.664465 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:40.664500 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:40.664556 23578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:40.664597 23578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:35:40.664654 23578 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:35:40.664691 23578 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39113/jwks.json
I20260929 16:35:40.673837 23578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:44395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39113/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:40.675078 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:40.676819 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:40.688158 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:40.688197 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:40.688122 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:40.689030 23578 server_base.cc:1034] running on GCE node
I20260929 16:35:40.689831 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:40.691319 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:40.692526 23578 hybrid_clock.cc:648] HybridClock initialized: now 1790699740692476 us; error 46 us; skew 500 ppm
I20260929 16:35:40.692849 23578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:40.698235 23578 webserver.cc:460] Webserver started at http://127.17.205.62:33925/ using document root <none> and password file <none>
I20260929 16:35:40.698944 23578 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:40.699051 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:40.699415 23578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:40.702316 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "4112f534f8294c95bc039eb4a35ea1f0"
format_stamp: "Formatted at 2026-09-29 16:35:40 on dist-test-slave-3h01"
server_key: "2377388faf4855aa444216ecc915987d"
server_key_iv: "2218e361180f214f9beaf0923d94d716"
server_key_version: "encryptionkey@0"
I20260929 16:35:40.703092 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "4112f534f8294c95bc039eb4a35ea1f0"
format_stamp: "Formatted at 2026-09-29 16:35:40 on dist-test-slave-3h01"
server_key: "2377388faf4855aa444216ecc915987d"
server_key_iv: "2218e361180f214f9beaf0923d94d716"
server_key_version: "encryptionkey@0"
I20260929 16:35:40.708243 23578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:35:40.711807 23592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:40.713243 23578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:35:40.713429 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "4112f534f8294c95bc039eb4a35ea1f0"
format_stamp: "Formatted at 2026-09-29 16:35:40 on dist-test-slave-3h01"
server_key: "2377388faf4855aa444216ecc915987d"
server_key_iv: "2218e361180f214f9beaf0923d94d716"
server_key_version: "encryptionkey@0"
I20260929 16:35:40.713639 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:40.730223 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:40.731096 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:40.731431 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:40.748361 23578 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:44395
I20260929 16:35:40.748395 23643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:44395 every 8 connection(s)
I20260929 16:35:40.749724 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:40.753539 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:40.754541 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23578
I20260929 16:35:40.754756 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:40.755539 18228 external_mini_cluster.cc:1428] Setting key 095d12a585627f806e683cc6e33fb257
I20260929 16:35:40.766669 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0: Bootstrap starting.
I20260929 16:35:40.768821 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:40.757575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.17.205.62:44395)
0929 16:35:40.758673 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:40.758685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:40.758730 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:35:40.760331 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:40.760360 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:40.760904 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:40.761278 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:40.761849 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.763098 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.764410 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.765113 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.765281 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.766418 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:40.766426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:40.766442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:40.767033 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:40.767122 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:40.767144 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:40.767274 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:40.767512 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:35:40.767820 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":391,"threads_started":1}
I20260929 16:35:40.770017 23644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:40.771065 23644 log.cc:824] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:40.772744 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:40.774060 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0: No bootstrap required, opened a new log
I20260929 16:35:40.781863 23644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } }
I20260929 16:35:40.782328 23644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:40.782437 23644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112f534f8294c95bc039eb4a35ea1f0, State: Initialized, Role: FOLLOWER
I20260929 16:35:40.783061 23644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } }
I20260929 16:35:40.783313 23644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:40.783445 23644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:40.783614 23644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:40.785710 23644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } }
I20260929 16:35:40.786238 23644 leader_election.cc:302] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4112f534f8294c95bc039eb4a35ea1f0; no voters:
I20260929 16:35:40.786819 23644 leader_election.cc:288] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:40.786960 23649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:40.788225 23649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [term 1 LEADER]: Becoming Leader. State: Replica: 4112f534f8294c95bc039eb4a35ea1f0, State: Running, Role: LEADER
I20260929 16:35:40.788761 23649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } }
I20260929 16:35:40.789353 23644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:40.791503 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4112f534f8294c95bc039eb4a35ea1f0. Latest consensus state: current_term: 1 leader_uuid: "4112f534f8294c95bc039eb4a35ea1f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } } }
I20260929 16:35:40.791882 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:40.792340 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4112f534f8294c95bc039eb4a35ea1f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112f534f8294c95bc039eb4a35ea1f0" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44395 } } }
I20260929 16:35:40.792649 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:40.793006 23658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:40.798456 23658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:40.807336 23658 catalog_manager.cc:1347] Generated new cluster ID: 1d2058a1e0e744e9ac2a82a340583e6e
I20260929 16:35:40.807467 23658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:40.821868 23658 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:40.823751 23658 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:40.841470 23658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4112f534f8294c95bc039eb4a35ea1f0: Generated new TSK 0
I20260929 16:35:40.842417 23658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:40.869347 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:40.861018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.17.205.62:44395)
0929 16:35:40.861213 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:40.861220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:40.861583 (+ 363us) server_negotiation.cc:407] Connection header received
0929 16:35:40.861881 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:40.861886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:40.861976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:40.862126 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:35:40.862678 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.863540 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.865539 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.865993 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.866105 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.866745 (+ 640us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:35:40.868697 (+ 1952us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:35:40.869031 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 16:35:40.869483 23647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:35:40.869581 23673 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:40.860826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44395 (local address 127.0.0.1:54976)
0929 16:35:40.861473 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:40.861492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:40.861648 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:40.862168 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:40.862173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:40.862187 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:35:40.862517 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:40.862530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.863673 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.863679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:40.865362 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:40.865381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:40.866192 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:40.866199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:40.866420 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:40.866473 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:35:40.868923 (+ 2450us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:35:40.869175 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.205.62:44395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":316,"threads_started":1}
I20260929 16:35:40.870419 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23578
I20260929 16:35:40.871555 23639 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 16:35:40.881116 23674 webserver.cc:533] Webserver: sslize failed:
E20260929 16:35:40.881234 23674 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:35:40.961349 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23578
2026-09-29T16:35:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:41Z Disabled control of system clock
I20260929 16:35:41.037009 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41467
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:41467
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/testchainca.cert with env {}
W20260929 16:35:41.380245 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:41.380558 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:41.380645 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:41.387804 23679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:41.387935 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:41.387972 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:41.388024 23679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:41.388067 23679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:35:41.388123 23679 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:35:41.388161 23679 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:35:41.398361 23679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:41467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:41.399791 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:41.401103 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:41.412497 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:41.412572 23685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:41.412598 23687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:41.413375 23679 server_base.cc:1034] running on GCE node
I20260929 16:35:41.414151 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:41.415647 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:41.416873 23679 hybrid_clock.cc:648] HybridClock initialized: now 1790699741416816 us; error 51 us; skew 500 ppm
I20260929 16:35:41.417227 23679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:35:41.419622 23679 webserver.cc:460] Webserver started at http://127.17.205.62:33219/ using document root <none> and password file <none>
I20260929 16:35:41.420303 23679 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:41.420403 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:41.420697 23679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:41.423552 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "39fecd06c0724c56a8a9a29dd4a37098"
format_stamp: "Formatted at 2026-09-29 16:35:41 on dist-test-slave-3h01"
server_key: "a04d07d36a73c6540a2fd44ecbaced87"
server_key_iv: "cb56ec50ea9f3a814ceee3dd90d1272d"
server_key_version: "encryptionkey@0"
I20260929 16:35:41.424319 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "39fecd06c0724c56a8a9a29dd4a37098"
format_stamp: "Formatted at 2026-09-29 16:35:41 on dist-test-slave-3h01"
server_key: "a04d07d36a73c6540a2fd44ecbaced87"
server_key_iv: "cb56ec50ea9f3a814ceee3dd90d1272d"
server_key_version: "encryptionkey@0"
I20260929 16:35:41.429417 23679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:41.432835 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:41.434351 23679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:35:41.434549 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "39fecd06c0724c56a8a9a29dd4a37098"
format_stamp: "Formatted at 2026-09-29 16:35:41 on dist-test-slave-3h01"
server_key: "a04d07d36a73c6540a2fd44ecbaced87"
server_key_iv: "cb56ec50ea9f3a814ceee3dd90d1272d"
server_key_version: "encryptionkey@0"
I20260929 16:35:41.434772 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:41.469503 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:41.470507 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:41.470824 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:41.488715 23679 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:41467
I20260929 16:35:41.488734 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:41467 every 8 connection(s)
I20260929 16:35:41.490159 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:41.490532 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23679
I20260929 16:35:41.490682 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:41.491482 18228 external_mini_cluster.cc:1428] Setting key 8a672df94059ec7e2005fe64e186c7ad
I20260929 16:35:41.494886 23745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:41.507865 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:41.493501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.17.205.62:41467)
0929 16:35:41.494601 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:41.494617 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:41.494672 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:35:41.496996 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:41.497025 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:41.497638 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:41.498140 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:41.498788 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.500870 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.502287 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.503398 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.503606 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:41.504693 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:41.504704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:41.504725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:41.505528 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:41.505630 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:41.505660 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:41.505804 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:41.505998 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:35:41.506376 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260929 16:35:41.509521 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098: Bootstrap starting.
I20260929 16:35:41.512014 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:41.513326 23745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:41.514546 23745 log.cc:824] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:41.517490 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098: No bootstrap required, opened a new log
I20260929 16:35:41.525445 23745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } }
I20260929 16:35:41.525955 23745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:41.526075 23745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fecd06c0724c56a8a9a29dd4a37098, State: Initialized, Role: FOLLOWER
I20260929 16:35:41.526686 23745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } }
I20260929 16:35:41.526912 23745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:41.527017 23745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:41.527153 23745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:41.529449 23745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } }
I20260929 16:35:41.529960 23745 leader_election.cc:302] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39fecd06c0724c56a8a9a29dd4a37098; no voters:
I20260929 16:35:41.530556 23745 leader_election.cc:288] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:41.530668 23750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:41.531332 23750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [term 1 LEADER]: Becoming Leader. State: Replica: 39fecd06c0724c56a8a9a29dd4a37098, State: Running, Role: LEADER
I20260929 16:35:41.532181 23750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } }
I20260929 16:35:41.532974 23745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:41.536113 23752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39fecd06c0724c56a8a9a29dd4a37098. Latest consensus state: current_term: 1 leader_uuid: "39fecd06c0724c56a8a9a29dd4a37098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } } }
I20260929 16:35:41.536525 23752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:41.543565 23751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39fecd06c0724c56a8a9a29dd4a37098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fecd06c0724c56a8a9a29dd4a37098" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41467 } } }
I20260929 16:35:41.544562 23751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:41.552225 23757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:41.557678 23757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:41.569768 23757 catalog_manager.cc:1347] Generated new cluster ID: 98ae179a7a994968afde8d9a4881ade6
I20260929 16:35:41.569911 23757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:41.582706 23757 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:41.584568 23757 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:41.596099 23757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39fecd06c0724c56a8a9a29dd4a37098: Generated new TSK 0
I20260929 16:35:41.597188 23757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:42.101688 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:41.620405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.17.205.62:41467)
0929 16:35:41.620617 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:41.620623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:41.620925 (+ 302us) server_negotiation.cc:407] Connection header received
0929 16:35:41.621131 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:41.621137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:41.621234 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:41.621404 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:35:41.621764 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.622861 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.624286 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.624984 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.625121 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:41.625594 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:35:42.101151 (+475557us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:35:42.101475 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45134: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 16:35:42.101787 23748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45134: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 16:35:42.102105 23774 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:41.620263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41467 (local address 127.0.0.1:45134)
0929 16:35:41.620823 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:41.620839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:41.620960 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:41.621403 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:41.621406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:41.621414 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:35:41.621627 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:41.621635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.623021 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.623026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:41.624142 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:41.624156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:41.625142 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:41.625146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:41.625275 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:41.625321 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:35:42.101437 (+476116us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:35:42.101678 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.205.62:41467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":287,"threads_started":1}
I20260929 16:35:42.103837 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23679
I20260929 16:35:42.104849 23740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:42.203817 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23679
2026-09-29T16:35:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1223 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1386 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:44 dist-test-slave-3h01 krb5kdc[23790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:44 dist-test-slave-3h01 krb5kdc[23790](info): set up 2 sockets
Sep 29 16:35:44 dist-test-slave-3h01 krb5kdc[23790](info): commencing operation
krb5kdc: starting...
W20260929 16:35:44.995662 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:45 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:35:45.192386 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:38071
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:45.508960 23803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:45.509271 23803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:45.509409 23803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:45.516517 23803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:45.516630 23803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:45.516674 23803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:45.516737 23803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:45.516786 23803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:45.526788 23803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:38071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:45.528096 23803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:45.529356 23803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:45.540601 23808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:45.540642 23809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:45.540663 23811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:45.541668 23803 server_base.cc:1034] running on GCE node
I20260929 16:35:45.542469 23803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:45.543901 23803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:45.545109 23803 hybrid_clock.cc:648] HybridClock initialized: now 1790699745545063 us; error 31 us; skew 500 ppm
I20260929 16:35:45.545442 23803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:45 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699745, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:45.548508 23803 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:35:45.550007 23803 webserver.cc:460] Webserver started at http://127.17.205.62:37153/ using document root <none> and password file <none>
I20260929 16:35:45.550674 23803 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:45.550789 23803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:45.551085 23803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:45.554003 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa"
format_stamp: "Formatted at 2026-09-29 16:35:45 on dist-test-slave-3h01"
server_key: "1760b820503fff7eaea412d12f6a46c4"
server_key_iv: "13a363ebf8304649c519cb778cf32f51"
server_key_version: "encryptionkey@0"
I20260929 16:35:45.554719 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa"
format_stamp: "Formatted at 2026-09-29 16:35:45 on dist-test-slave-3h01"
server_key: "1760b820503fff7eaea412d12f6a46c4"
server_key_iv: "13a363ebf8304649c519cb778cf32f51"
server_key_version: "encryptionkey@0"
I20260929 16:35:45.559614 23803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:35:45.562985 23818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:45.564317 23803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:45.564476 23803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa"
format_stamp: "Formatted at 2026-09-29 16:35:45 on dist-test-slave-3h01"
server_key: "1760b820503fff7eaea412d12f6a46c4"
server_key_iv: "13a363ebf8304649c519cb778cf32f51"
server_key_version: "encryptionkey@0"
I20260929 16:35:45.564656 23803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:45.601089 23803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:45.604980 23803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:45.605238 23803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:45.621919 23803 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:38071
I20260929 16:35:45.621932 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:38071 every 8 connection(s)
I20260929 16:35:45.623301 23803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:45.626964 23870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:45.631590 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23803
I20260929 16:35:45.631780 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:45.632364 18228 external_mini_cluster.cc:1428] Setting key 3d4a920a7a15d554848e38fb05406cee
I20260929 16:35:45.637354 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa: Bootstrap starting.
I20260929 16:35:45.640933 23870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:45.642165 23870 log.cc:824] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:45.645519 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa: No bootstrap required, opened a new log
Sep 29 16:35:45 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:45.653970 23870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } }
I20260929 16:35:45.654349 23870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:45.654385 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:45.634173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.17.205.62:38071)
0929 16:35:45.634916 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:45.634928 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:45.634977 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:35:45.636676 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:45.636701 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:45.637169 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:45.637508 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:45.639127 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:45.640436 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:45.642131 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:45.642998 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:45.643178 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:45.647598 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:45.647606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:45.647621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:45.647658 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:45.650490 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:45.651252 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:45.651261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:45.651267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:45.651435 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:45.651786 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:45.651789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:45.651791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:45.652327 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:45.652552 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:45.652988 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:35:45.653354 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":310,"threads_started":1}
I20260929 16:35:45.654451 23870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ed658aa2d7481cbf17d7f4c7e7dbaa, State: Initialized, Role: FOLLOWER
I20260929 16:35:45.655138 23870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } }
I20260929 16:35:45.655443 23870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:45.655576 23870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:45.655740 23870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:45.658074 23870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } }
I20260929 16:35:45.658525 23870 leader_election.cc:302] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ed658aa2d7481cbf17d7f4c7e7dbaa; no voters:
I20260929 16:35:45.659011 23870 leader_election.cc:288] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:45.659130 23875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:45.660048 23875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [term 1 LEADER]: Becoming Leader. State: Replica: 33ed658aa2d7481cbf17d7f4c7e7dbaa, State: Running, Role: LEADER
I20260929 16:35:45.660771 23875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } }
I20260929 16:35:45.661032 23870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:45.663355 23877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ed658aa2d7481cbf17d7f4c7e7dbaa. Latest consensus state: current_term: 1 leader_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } } }
I20260929 16:35:45.663702 23877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:45.664153 23876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ed658aa2d7481cbf17d7f4c7e7dbaa" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38071 } } }
I20260929 16:35:45.664419 23876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:45.665449 23883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:45.677336 23883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:45.686638 23883 catalog_manager.cc:1347] Generated new cluster ID: 3904fa2aa5884138a5c95fedb93187bd
I20260929 16:35:45.686762 23883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:45.727286 23883 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:45.728642 23883 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:45.742579 23883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ed658aa2d7481cbf17d7f4c7e7dbaa: Generated new TSK 0
I20260929 16:35:45.743467 23883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:35:45.799940 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:46.127388 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:46.127704 23896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:46.127846 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:46.135154 23896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:46.135339 23896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:46.135504 23896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:46.145766 23896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:46.147140 23896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:46.148573 23896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:46.161787 23901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:46.161829 23902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:46.162058 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:46.162606 23896 server_base.cc:1034] running on GCE node
I20260929 16:35:46.163329 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:46.164302 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:46.165522 23896 hybrid_clock.cc:648] HybridClock initialized: now 1790699746165466 us; error 51 us; skew 500 ppm
I20260929 16:35:46.165853 23896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:46 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699746, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:46.173135 23896 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:35:46.174852 23896 webserver.cc:460] Webserver started at http://127.17.205.1:42883/ using document root <none> and password file <none>
I20260929 16:35:46.175704 23896 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:46.175822 23896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:46.176111 23896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:46.179179 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "f3e0bf91be454623af7e1bdab5697a1f"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "e38aa5b7b525b0ce7aa5eecf93ba598e"
server_key_iv: "9906c4d0f903b51064c733605d4461e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.180063 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "f3e0bf91be454623af7e1bdab5697a1f"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "e38aa5b7b525b0ce7aa5eecf93ba598e"
server_key_iv: "9906c4d0f903b51064c733605d4461e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.185889 23896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:46.189947 23911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.191586 23896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:35:46.191753 23896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "f3e0bf91be454623af7e1bdab5697a1f"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "e38aa5b7b525b0ce7aa5eecf93ba598e"
server_key_iv: "9906c4d0f903b51064c733605d4461e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.191962 23896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:46.203883 23896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:46.208667 23896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:46.208953 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:46.209939 23896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:46.211426 23896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:46.211498 23896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.211611 23896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:46.211665 23896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.241544 23896 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:43615
I20260929 16:35:46.241600 24023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:43615 every 8 connection(s)
I20260929 16:35:46.242928 23896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:46.251726 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 23896
I20260929 16:35:46.251899 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:46.252508 18228 external_mini_cluster.cc:1428] Setting key c9a08f9d9f0f9ae4508fc4e5b99073a4
Sep 29 16:35:46 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699746, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:46.263002 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:46.245415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:39157 (local address 127.17.205.62:38071)
0929 16:35:46.245807 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:46.245812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:46.246840 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 16:35:46.248607 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:46.248611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:46.248678 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:46.248775 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:46.250221 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.250921 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.252591 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.253019 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.253156 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:46.256785 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:46.256791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:46.256796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:46.256832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:46.259721 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:46.260502 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:46.260508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:46.260512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:46.260674 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:46.261266 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:46.261269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:46.261271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:46.261552 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:46.261660 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:46.262561 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 16:35:46.262738 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":205,"threads_started":1}
I20260929 16:35:46.264417 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:46.246001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38071 (local address 127.17.205.1:39157)
0929 16:35:46.246683 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:46.246728 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:46.248350 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:46.248988 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:46.248996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:46.249521 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:46.250090 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:46.250108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.251034 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.251038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:46.252395 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:46.252424 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.253240 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.253248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:46.253529 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:46.254145 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:46.254167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:46.256567 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:46.259908 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:46.259917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:46.259931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:46.260379 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:46.260852 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:46.260858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:46.260886 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:46.261159 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:46.261678 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:46.261686 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:46.262140 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 16:35:46.262432 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 16:35:46.263006 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260929 16:35:46.266000 24024 heartbeater.cc:347] Connected to a master server at 127.17.205.62:38071
I20260929 16:35:46.266311 24024 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:46.267027 24024 heartbeater.cc:511] Master 127.17.205.62:38071 requested a full tablet report, sending...
I20260929 16:35:46.269313 23835 ts_manager.cc:194] Registered new tserver with Master: f3e0bf91be454623af7e1bdab5697a1f (127.17.205.1:43615)
I20260929 16:35:46.270748 23835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:39157
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:35:46.298884 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:46.612780 24030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:46.613088 24030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:46.613227 24030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:46.620636 24030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:46.620770 24030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:46.620918 24030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:46.630826 24030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:46.632164 24030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:46.633512 24030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:46.646142 24036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:46.646255 24035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:46.646185 24038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:46.646845 24030 server_base.cc:1034] running on GCE node
I20260929 16:35:46.647500 24030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:46.648444 24030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:46.649641 24030 hybrid_clock.cc:648] HybridClock initialized: now 1790699746649587 us; error 49 us; skew 500 ppm
I20260929 16:35:46.649945 24030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:46 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699746, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:46.653061 24030 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:35:46.654538 24030 webserver.cc:460] Webserver started at http://127.17.205.2:37947/ using document root <none> and password file <none>
I20260929 16:35:46.655269 24030 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:46.655387 24030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:46.655702 24030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:46.658579 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "9416de8027234074b9b2789b1c4b6a35"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "35e5937025174e4123f8c46d1ca5fd6b"
server_key_iv: "f44a98823a4f56f42a3ef85087a2282f"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.659385 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "9416de8027234074b9b2789b1c4b6a35"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "35e5937025174e4123f8c46d1ca5fd6b"
server_key_iv: "f44a98823a4f56f42a3ef85087a2282f"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.664688 24030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:35:46.668157 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.669569 24030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:46.669721 24030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "9416de8027234074b9b2789b1c4b6a35"
format_stamp: "Formatted at 2026-09-29 16:35:46 on dist-test-slave-3h01"
server_key: "35e5937025174e4123f8c46d1ca5fd6b"
server_key_iv: "f44a98823a4f56f42a3ef85087a2282f"
server_key_version: "encryptionkey@0"
I20260929 16:35:46.669917 24030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:46.726791 24030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:46.731352 24030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:46.731624 24030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:46.732589 24030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:46.733954 24030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:46.734032 24030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.734149 24030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:46.734200 24030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:46.763733 24030 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:40687
I20260929 16:35:46.763833 24157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:40687 every 8 connection(s)
I20260929 16:35:46.765045 24030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:46.767899 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24030
I20260929 16:35:46.768060 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:46.768463 18228 external_mini_cluster.cc:1428] Setting key 1fcfb95a0f3d646b09d2ee47368fd741
Sep 29 16:35:46 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699746, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:46.787438 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:46.767678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33155 (local address 127.17.205.62:38071)
0929 16:35:46.768107 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:46.768113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:46.769393 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 16:35:46.771602 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:46.771606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:46.771672 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:46.771759 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:46.773860 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.774619 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.776740 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.777253 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.777372 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:46.781058 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:46.781065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:46.781069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:46.781107 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:46.784055 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:46.784830 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:46.784835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:46.784844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:46.784992 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:46.785476 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:46.785482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:46.785485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:46.785853 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:46.786006 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:46.786771 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 16:35:46.787024 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":228,"threads_started":1}
I20260929 16:35:46.788246 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:46.768348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38071 (local address 127.17.205.2:33155)
0929 16:35:46.769210 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:46.769275 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:46.771306 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:46.772056 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:46.772069 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:46.772836 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:46.773653 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:46.773678 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.774753 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.774758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:46.776593 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:46.776613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:46.777475 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:46.777480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:46.777683 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:46.778430 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:46.778455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:46.780784 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:46.784215 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:46.784223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:46.784239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:46.784696 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:46.785087 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:46.785090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:46.785092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:46.785346 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:46.785991 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:46.785998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:46.786374 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 16:35:46.786617 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 16:35:46.787047 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:35:46.789793 24158 heartbeater.cc:347] Connected to a master server at 127.17.205.62:38071
I20260929 16:35:46.790115 24158 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:46.790861 24158 heartbeater.cc:511] Master 127.17.205.62:38071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:35:46.792392 23835 ts_manager.cc:194] Registered new tserver with Master: 9416de8027234074b9b2789b1c4b6a35 (127.17.205.2:40687)
I20260929 16:35:46.793175 23835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:33155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:35:46.815609 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:47.130335 24164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:47.130602 24164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:47.130702 24164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:47.137826 24164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:47.137967 24164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:47.138084 24164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:47.147656 24164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:47.148878 24164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:47.150118 24164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:47.162734 24169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:47.162735 24170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:47.162901 24172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:47.163468 24164 server_base.cc:1034] running on GCE node
I20260929 16:35:47.164041 24164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:47.164963 24164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:47.166155 24164 hybrid_clock.cc:648] HybridClock initialized: now 1790699747166102 us; error 52 us; skew 500 ppm
I20260929 16:35:47.166446 24164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.169436 24164 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:35:47.170848 24164 webserver.cc:460] Webserver started at http://127.17.205.3:41525/ using document root <none> and password file <none>
I20260929 16:35:47.171623 24164 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:47.171736 24164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:47.172037 24164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:47.175019 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "9ffc3802e9a245be8c89a2352164d092"
format_stamp: "Formatted at 2026-09-29 16:35:47 on dist-test-slave-3h01"
server_key: "f7ca1b7cc366a1f167ea333e36c05991"
server_key_iv: "27d8dcd939781192384be3eefcd84403"
server_key_version: "encryptionkey@0"
I20260929 16:35:47.175773 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffc3802e9a245be8c89a2352164d092"
format_stamp: "Formatted at 2026-09-29 16:35:47 on dist-test-slave-3h01"
server_key: "f7ca1b7cc366a1f167ea333e36c05991"
server_key_iv: "27d8dcd939781192384be3eefcd84403"
server_key_version: "encryptionkey@0"
I20260929 16:35:47.180764 24164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:47.184379 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:47.185804 24164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:47.185981 24164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "9ffc3802e9a245be8c89a2352164d092"
format_stamp: "Formatted at 2026-09-29 16:35:47 on dist-test-slave-3h01"
server_key: "f7ca1b7cc366a1f167ea333e36c05991"
server_key_iv: "27d8dcd939781192384be3eefcd84403"
server_key_version: "encryptionkey@0"
I20260929 16:35:47.186179 24164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:47.218888 24164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:47.223061 24164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:47.223367 24164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:47.224308 24164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:47.225620 24164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:47.225695 24164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:47.225821 24164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:47.225872 24164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:47.255645 24164 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43023
I20260929 16:35:47.255688 24291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43023 every 8 connection(s)
I20260929 16:35:47.256875 24164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:47.264519 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24164
I20260929 16:35:47.264707 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:47.265167 18228 external_mini_cluster.cc:1428] Setting key dde03156e94c8bdb4dc019141cea73bb
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:35:47.273893 24024 heartbeater.cc:503] Master 127.17.205.62:38071 was elected leader, sending a full tablet report...
I20260929 16:35:47.277014 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.259343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:34415 (local address 127.17.205.62:38071)
0929 16:35:47.259503 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.259507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.260740 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 16:35:47.262643 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.262647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.262718 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.262802 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.264555 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.265388 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.267195 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.267625 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.267718 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.271435 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.271440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.271444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.271496 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.274060 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.274882 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.274887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.274891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.275017 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.275472 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.275476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.275478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.275749 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.275876 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.276576 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.276757 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:35:47.278093 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.259800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38071 (local address 127.17.205.3:34415)
0929 16:35:47.260543 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:47.260599 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:47.262365 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:47.263059 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:47.263068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:47.263665 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:47.264376 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:47.264404 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.265550 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.265556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:47.267066 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:47.267079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.267893 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.267901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:47.268171 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.268859 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:47.268888 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:47.271166 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:47.274236 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:47.274244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:47.274257 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:47.274757 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:47.275117 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:47.275122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:47.275125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:47.275352 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:47.275850 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:47.275863 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:47.276187 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 16:35:47.276417 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:35:47.276806 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":299,"threads_started":1}
I20260929 16:35:47.279975 24292 heartbeater.cc:347] Connected to a master server at 127.17.205.62:38071
I20260929 16:35:47.280450 24292 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:47.281409 24292 heartbeater.cc:511] Master 127.17.205.62:38071 requested a full tablet report, sending...
I20260929 16:35:47.282785 23835 ts_manager.cc:194] Registered new tserver with Master: 9ffc3802e9a245be8c89a2352164d092 (127.17.205.3:43023)
I20260929 16:35:47.283628 23835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:34415
I20260929 16:35:47.293658 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/test-admin.keytab.
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.318404 18228 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:35:47.323258 24299 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:47.321004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.17.205.1:43615)
0929 16:35:47.321589 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.321596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.321622 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:35:47.321683 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.321690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.321908 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.322061 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.322857 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35136: BlockingRecv error: recv got EOF from 127.0.0.1:35136 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
W20260929 16:35:47.323567 18228 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc)
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.325172 18228 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.326483 18228 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:47.341252 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.328024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.17.205.1:43615)
0929 16:35:47.328205 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.328212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.328458 (+ 246us) server_negotiation.cc:407] Connection header received
0929 16:35:47.328586 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.328596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.328656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.328735 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.329723 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.330680 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.332012 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.332606 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.332711 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.336117 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.336128 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.336144 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.336185 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.338755 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.339409 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.339417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.339421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.339545 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.339827 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.339830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.339832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.340177 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.340444 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.340737 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.340998 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 16:35:47.346854 24299 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:47.345246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.17.205.1:43615)
0929 16:35:47.345457 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.345463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.345604 (+ 141us) server_negotiation.cc:407] Connection header received
0929 16:35:47.345730 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.345734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.345799 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.345877 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.346631 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:35:47.347162 18228 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.348690 18228 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.349915 18228 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:47.363755 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.351141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.17.205.1:43615)
0929 16:35:47.351334 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.351340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.351612 (+ 272us) server_negotiation.cc:407] Connection header received
0929 16:35:47.351768 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.351773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.351864 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.351943 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.352883 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.353567 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.354740 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.355176 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.355352 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.358804 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.358809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.358812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.358875 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.361377 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.362001 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.362007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.362009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.362124 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.362481 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.362486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.362489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.362754 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.363001 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.363291 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.363490 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:35:47.367820 24299 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:47.366119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.17.205.1:43615)
0929 16:35:47.366305 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.366311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.366525 (+ 214us) server_negotiation.cc:407] Connection header received
0929 16:35:47.366675 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.366679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.366751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.366827 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.367627 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35154: BlockingRecv error: recv got EOF from 127.0.0.1:35154 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 16:35:47.368038 18228 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699695913193-18228-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.369639 18228 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:47.370806 18228 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:35:47.384362 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.372101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.17.205.1:43615)
0929 16:35:47.372241 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.372245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.372630 (+ 385us) server_negotiation.cc:407] Connection header received
0929 16:35:47.372775 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.372780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.372859 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.372943 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.373922 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.374700 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.375901 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.376325 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.376430 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.379684 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.379689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.379692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.379726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.381955 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.382561 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.382565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.382567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.382701 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.383033 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.383036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.383039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.383329 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.383606 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.383869 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.384077 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:35:47.385797 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23896
I20260929 16:35:47.396917 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.386505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.17.205.1:43615)
0929 16:35:47.386641 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.386645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.386879 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:35:47.387012 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.387016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.387069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.387134 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.388030 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.388692 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.389920 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.390376 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.390469 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.392363 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.392368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.392371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.392400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.394607 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.395153 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.395156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.395158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.395327 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.395679 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.395683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.395685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.395978 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.396166 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.396424 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.396582 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:35:47.397487 24019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:47.478950 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23896
I20260929 16:35:47.491998 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24030
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:35:47.506692 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.492856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.17.205.2:40687)
0929 16:35:47.493318 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.493325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.493348 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:35:47.493404 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.493411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.493632 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.493773 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.494706 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.495739 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.496959 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.497513 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.497625 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.501272 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.501284 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.501299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.501336 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.504008 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.504674 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.504682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.504685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.504805 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.505169 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.505173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.505175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.505532 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.505819 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.506144 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.506391 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":212,"threads_started":1}
I20260929 16:35:47.508172 24153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:47.589346 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24030
I20260929 16:35:47.601727 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24164
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[23790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:35:47.617007 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:47.602678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.17.205.3:43023)
0929 16:35:47.603317 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:47.603325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:47.603356 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:47.603416 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:47.603423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:47.603630 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:47.603783 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:47.604786 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.605847 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.607040 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:47.607727 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:47.607868 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:47.611687 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:47.611700 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:47.611714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:47.611745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:47.614355 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.615004 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.615011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.615013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.615135 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:47.615589 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:47.615594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:47.615598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:47.615934 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:47.616219 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:47.616471 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:35:47.616714 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260929 16:35:47.618218 24287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:47.706864 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24164
I20260929 16:35:47.719770 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 23803
I20260929 16:35:47.720870 23865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:47.822357 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 23803
2026-09-29T16:35:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3834 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[24319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[24319](info): set up 2 sockets
Sep 29 16:35:47 dist-test-slave-3h01 krb5kdc[24319](info): commencing operation
krb5kdc: starting...
W20260929 16:35:48.823804 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:35:48 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab.
Entry for principal oryx/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab.
I20260929 16:35:49.023300 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:46139
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab
--principal=oryx/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:49.341787 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:49.342087 24332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:49.342185 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:49.349717 24332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:49.349862 24332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:49.349916 24332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:49.349980 24332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:49.350029 24332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:49.360140 24332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab
--principal=oryx/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:49.361498 24332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:49.362811 24332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:49.374368 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:49.374354 24338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:49.374497 24340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:49.375562 24332 server_base.cc:1034] running on GCE node
I20260929 16:35:49.376324 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:49.377774 24332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:49.378976 24332 hybrid_clock.cc:648] HybridClock initialized: now 1790699749378922 us; error 41 us; skew 500 ppm
I20260929 16:35:49.379359 24332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:49 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699749, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:49.382380 24332 init.cc:377] Logged in from keytab as oryx/127.17.205.62@KRBTEST.COM (short username oryx)
I20260929 16:35:49.383919 24332 webserver.cc:460] Webserver started at http://127.17.205.62:45687/ using document root <none> and password file <none>
I20260929 16:35:49.384593 24332 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:49.384703 24332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:49.384977 24332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:49.387916 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "6f6e48515165498ea244e5bc2e35e080"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "0cfcac5040e7d60357c5717c6c2f1173"
server_key_iv: "01357045a0262b48a62a5d631a99f7e5"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.388656 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "6f6e48515165498ea244e5bc2e35e080"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "0cfcac5040e7d60357c5717c6c2f1173"
server_key_iv: "01357045a0262b48a62a5d631a99f7e5"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.393750 24332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:49.397076 24347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:49.398361 24332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:49.398515 24332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6f6e48515165498ea244e5bc2e35e080"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "0cfcac5040e7d60357c5717c6c2f1173"
server_key_iv: "01357045a0262b48a62a5d631a99f7e5"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.398694 24332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:49.416923 24332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:49.420720 24332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:49.420993 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:49.437194 24332 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:46139
I20260929 16:35:49.437212 24398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:46139 every 8 connection(s)
I20260929 16:35:49.438527 24332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:49.442226 24399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:49.442266 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24332
I20260929 16:35:49.442441 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:35:49.443051 18228 external_mini_cluster.cc:1428] Setting key 26d6867a6acdfc297def5b5646053b59
I20260929 16:35:49.455899 24399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Bootstrap starting.
Sep 29 16:35:49 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:49.458845 24399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:49.460172 24399 log.cc:824] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:49.463524 24399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: No bootstrap required, opened a new log
I20260929 16:35:49.467032 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:49.444977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.17.205.62:46139)
0929 16:35:49.445770 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:49.445785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:49.445833 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:35:49.448440 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:49.448469 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:49.449008 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:49.449462 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:49.450654 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:49.452577 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:49.453892 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:49.454901 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:49.455079 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:49.458959 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:49.458971 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:49.458990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:49.459038 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:49.462673 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:49.463405 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:49.463413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:49.463420 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:49.463625 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:49.463993 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:49.463998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:49.464000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:49.464671 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:49.464963 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:49.465216 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:35:49.465658 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":314,"threads_started":1}
I20260929 16:35:49.472427 24399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } }
I20260929 16:35:49.472836 24399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:49.472970 24399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6e48515165498ea244e5bc2e35e080, State: Initialized, Role: FOLLOWER
I20260929 16:35:49.473546 24399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } }
I20260929 16:35:49.473752 24399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:49.473868 24399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:49.474022 24399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:49.476140 24399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } }
I20260929 16:35:49.476612 24399 leader_election.cc:302] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6e48515165498ea244e5bc2e35e080; no voters:
I20260929 16:35:49.477159 24399 leader_election.cc:288] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:49.477298 24404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:49.478096 24404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 LEADER]: Becoming Leader. State: Replica: 6f6e48515165498ea244e5bc2e35e080, State: Running, Role: LEADER
I20260929 16:35:49.478732 24404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } }
I20260929 16:35:49.479193 24399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:49.481549 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } }
I20260929 16:35:49.481952 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:49.482338 24406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6e48515165498ea244e5bc2e35e080. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } }
I20260929 16:35:49.482692 24406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:49.482926 24413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:49.488579 24413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:49.498219 24413 catalog_manager.cc:1347] Generated new cluster ID: a879cd01851c48db81d9e05bc6417677
I20260929 16:35:49.498335 24413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:49.512954 24413 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:49.514451 24413 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:49.519685 24413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Generated new TSK 0
I20260929 16:35:49.520509 24413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
I20260929 16:35:49.574846 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:49.914811 24425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:49.915171 24425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:49.915315 24425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:49.922535 24425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:49.922659 24425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:49.922770 24425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:49.932523 24425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:49.933790 24425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:49.935096 24425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:49.947778 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:49.947816 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:49.947840 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:49.948628 24425 server_base.cc:1034] running on GCE node
I20260929 16:35:49.949210 24425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:49.950094 24425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:49.951305 24425 hybrid_clock.cc:648] HybridClock initialized: now 1790699749951257 us; error 34 us; skew 500 ppm
I20260929 16:35:49.951612 24425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:49 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699749, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:49.954571 24425 init.cc:377] Logged in from keytab as oryx/127.17.205.1@KRBTEST.COM (short username oryx)
I20260929 16:35:49.956034 24425 webserver.cc:460] Webserver started at http://127.17.205.1:34879/ using document root <none> and password file <none>
I20260929 16:35:49.956701 24425 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:49.956816 24425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:49.957103 24425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:49.960057 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "521cae2391a2cd1ac619549d88d34e4a"
server_key_iv: "bcb90da5b06a60c38eb096c54f1cefe9"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.960798 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "521cae2391a2cd1ac619549d88d34e4a"
server_key_iv: "bcb90da5b06a60c38eb096c54f1cefe9"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.965729 24425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:49.969254 24440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:49.970700 24425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:49.970842 24425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "521cae2391a2cd1ac619549d88d34e4a"
server_key_iv: "bcb90da5b06a60c38eb096c54f1cefe9"
server_key_version: "encryptionkey@0"
I20260929 16:35:49.971035 24425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:49.994835 24425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:49.999085 24425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:49.999402 24425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:50.000396 24425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:50.001747 24425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:50.001823 24425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.001946 24425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:50.002038 24425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 16:35:50.030573 24425 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:35643
I20260929 16:35:50.030619 24552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:35643 every 8 connection(s)
I20260929 16:35:50.031900 24425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:50.037472 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24425
I20260929 16:35:50.037680 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:35:50.038436 18228 external_mini_cluster.cc:1428] Setting key 78368409bb88e730ec337eb7a2f96460
Sep 29 16:35:50 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699749, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:50.052917 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.034447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:52913 (local address 127.17.205.62:46139)
0929 16:35:50.034860 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:50.034864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:50.035909 (+ 1045us) server_negotiation.cc:407] Connection header received
0929 16:35:50.038037 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:50.038041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:50.038115 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:50.038210 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:50.040193 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.040945 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.043141 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.043565 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.043663 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.047577 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:50.047581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:50.047585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:50.047615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:50.049946 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.050776 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.050780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.050783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.050902 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.051341 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.051344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.051346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.051604 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:50.051708 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:50.052497 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 16:35:50.052656 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":202,"threads_started":1}
I20260929 16:35:50.054291 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.035039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:52913)
0929 16:35:50.035754 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:50.035799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:50.037710 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:50.038502 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:50.038515 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:50.039238 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:50.040017 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.040044 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.041123 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.041128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.043011 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.043025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.043775 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.043781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.044085 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.044764 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:50.044792 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:50.047355 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:50.050091 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.050101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.050118 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.050661 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.050993 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.050997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.051001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.051248 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.051722 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:50.051730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:50.052149 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 16:35:50.052429 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:35:50.052944 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":309,"threads_started":1}
I20260929 16:35:50.055821 24553 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:50.056182 24553 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:50.057080 24553 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:35:50.059204 24364 ts_manager.cc:194] Registered new tserver with Master: 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:50.060691 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:52913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
I20260929 16:35:50.084271 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:50.397161 24559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:50.397449 24559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:50.397540 24559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:50.404873 24559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:50.405023 24559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:50.405145 24559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:50.414816 24559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:50.416109 24559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:50.417402 24559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:50.430475 24565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:50.430485 24564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:50.430513 24567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:50.431074 24559 server_base.cc:1034] running on GCE node
I20260929 16:35:50.431802 24559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:50.432775 24559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:50.433969 24559 hybrid_clock.cc:648] HybridClock initialized: now 1790699750433911 us; error 55 us; skew 500 ppm
I20260929 16:35:50.434268 24559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:50 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699750, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:50.437422 24559 init.cc:377] Logged in from keytab as oryx/127.17.205.2@KRBTEST.COM (short username oryx)
I20260929 16:35:50.438997 24559 webserver.cc:460] Webserver started at http://127.17.205.2:41907/ using document root <none> and password file <none>
I20260929 16:35:50.439699 24559 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:50.439801 24559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:50.440080 24559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:50.443003 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "5e8e51cda88e4922995232ba64e5172e"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "af26932385f8b5ea683e33c08a3e68d9"
server_key_iv: "68e415b4a97faf088070670c86dbafd4"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.443817 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "5e8e51cda88e4922995232ba64e5172e"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "af26932385f8b5ea683e33c08a3e68d9"
server_key_iv: "68e415b4a97faf088070670c86dbafd4"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.449179 24559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:50.452729 24574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.454038 24559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:50.454180 24559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "5e8e51cda88e4922995232ba64e5172e"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "af26932385f8b5ea683e33c08a3e68d9"
server_key_iv: "68e415b4a97faf088070670c86dbafd4"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.454355 24559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:50.464865 24559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:50.469028 24559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:50.469295 24559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:50.470240 24559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:50.471645 24559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:50.471727 24559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.471843 24559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:50.471889 24559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.500730 24559 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45241
I20260929 16:35:50.500770 24686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45241 every 8 connection(s)
I20260929 16:35:50.502007 24559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:50.502585 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24559
I20260929 16:35:50.502765 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:35:50.503372 18228 external_mini_cluster.cc:1428] Setting key 850cb909afd29fc0421419eaa01442f3
Sep 29 16:35:50 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699750, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:35:50.523387 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.504772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:35083 (local address 127.17.205.62:46139)
0929 16:35:50.504950 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:50.504955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:50.506265 (+ 1310us) server_negotiation.cc:407] Connection header received
0929 16:35:50.508062 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:50.508067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:50.508144 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:50.508238 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:50.509650 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.510636 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.512875 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.513295 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.513414 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.517192 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:50.517198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:50.517201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:50.517235 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:50.519927 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.520961 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.520965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.520968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.521107 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.521580 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.521583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.521585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.521836 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:50.521967 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:50.522927 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 16:35:50.523107 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:35:50.524423 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.505298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:35083)
0929 16:35:50.506098 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:50.506153 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:50.507821 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:50.508424 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:50.508433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:50.508923 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:50.509501 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.509518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.510782 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.510788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.512709 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.512728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.513535 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.513542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.513795 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.514482 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:50.514507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:50.516959 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:50.520114 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.520125 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.520147 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.520776 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.521214 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.521220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.521224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.521484 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.521970 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:50.521978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:50.522451 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 16:35:50.522761 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 16:35:50.523320 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":307,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.3@KRBTEST.COM" created.
I20260929 16:35:50.525861 24687 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:50.526163 24687 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:50.526895 24687 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
I20260929 16:35:50.528371 24364 ts_manager.cc:194] Registered new tserver with Master: 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
I20260929 16:35:50.529129 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:35083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
I20260929 16:35:50.548424 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:50.864104 24692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:50.864390 24692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:50.864509 24692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:50.871811 24692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:50.871973 24692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:50.872117 24692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:50.882151 24692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:50.883533 24692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:50.884924 24692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:50.898118 24697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:50.898185 24700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:50.898162 24698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:50.898839 24692 server_base.cc:1034] running on GCE node
I20260929 16:35:50.899560 24692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:50.900490 24692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:50.901700 24692 hybrid_clock.cc:648] HybridClock initialized: now 1790699750901634 us; error 55 us; skew 500 ppm
I20260929 16:35:50.902004 24692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:50 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699750, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:50.905303 24692 init.cc:377] Logged in from keytab as oryx/127.17.205.3@KRBTEST.COM (short username oryx)
I20260929 16:35:50.906808 24692 webserver.cc:460] Webserver started at http://127.17.205.3:38825/ using document root <none> and password file <none>
I20260929 16:35:50.907557 24692 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:50.907662 24692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:50.907924 24692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:50.910789 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "9ee28e215f494ab193143e44161047e8"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "dd7dfe1449c06645d63f6870e72d6f8a"
server_key_iv: "7ca05e136f86a63dfa70e2224e7541e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.911581 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "9ee28e215f494ab193143e44161047e8"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "dd7dfe1449c06645d63f6870e72d6f8a"
server_key_iv: "7ca05e136f86a63dfa70e2224e7541e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.916702 24692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:35:50.920282 24707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.921675 24692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:50.921809 24692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "9ee28e215f494ab193143e44161047e8"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "dd7dfe1449c06645d63f6870e72d6f8a"
server_key_iv: "7ca05e136f86a63dfa70e2224e7541e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:50.921998 24692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:50.934085 24692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:50.938149 24692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:50.938424 24692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:50.939378 24692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:50.940713 24692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:50.940783 24692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.940869 24692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:50.940910 24692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:50.969692 24692 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:32901
I20260929 16:35:50.969730 24819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:32901 every 8 connection(s)
I20260929 16:35:50.970901 24692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:50.976526 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24692
I20260929 16:35:50.976704 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:35:50.977130 18228 external_mini_cluster.cc:1428] Setting key f757d43e63ea4c6ffc15425acd0745a0
Sep 29 16:35:50 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699750, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:50.990298 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.973397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:50831 (local address 127.17.205.62:46139)
0929 16:35:50.973551 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:50.973556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:50.974704 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 16:35:50.976576 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:50.976582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:50.976696 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:50.976785 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:50.978413 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.979573 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.981307 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.981738 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.981864 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.985210 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:50.985215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:50.985218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:50.985250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:50.987503 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.988165 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.988169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.988171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.988282 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:50.988717 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:50.988721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:50.988723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:50.989004 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:50.989114 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:50.989833 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 16:35:50.990016 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:35:50.991308 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:50.973866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:50831)
0929 16:35:50.974552 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:50.974595 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:50.976301 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:50.977040 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:50.977049 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:50.977578 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:50.978232 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.978250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.979722 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.979725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.981163 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:50.981179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:50.981989 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:50.981995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:50.982192 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:50.982831 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:50.982872 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:50.985009 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:50.987632 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.987639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.987653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.988078 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.988399 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:50.988403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:50.988405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:50.988613 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:50.989125 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:50.989130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:50.989452 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 16:35:50.989694 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 16:35:50.990097 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":253,"threads_started":1}
I20260929 16:35:50.992712 24820 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:50.993045 24820 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:50.993698 24820 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
I20260929 16:35:50.995040 24364 ts_manager.cc:194] Registered new tserver with Master: 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901)
I20260929 16:35:50.995821 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:50831
I20260929 16:35:51.005053 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.205.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.61.keytab.
Entry for principal oryx/127.17.205.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.61.keytab.
I20260929 16:35:51.050805 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:38359
--webserver_interface=127.17.205.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.61.keytab
--principal=oryx/127.17.205.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:46139,127.17.205.61:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:35:51.063696 24553 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
W20260929 16:35:51.371589 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:51.371908 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:51.372057 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:51.379676 24825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:51.379791 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:51.379877 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:51.379956 24825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:51.380021 24825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:51.390094 24825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:46139,127.17.205.61:38359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.61.keytab
--principal=oryx/127.17.205.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.205.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
W20260929 16:35:51.390484 24825 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.205.62:46139,127.17.205.61:38359'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:35:51.391583 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:51.392936 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:51.405390 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:51.405366 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:51.405434 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:51.406088 24825 server_base.cc:1034] running on GCE node
I20260929 16:35:51.406862 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:51.408459 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:51.409679 24825 hybrid_clock.cc:648] HybridClock initialized: now 1790699751409614 us; error 40 us; skew 500 ppm
I20260929 16:35:51.409998 24825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699751, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:51.419144 24825 init.cc:377] Logged in from keytab as oryx/127.17.205.61@KRBTEST.COM (short username oryx)
I20260929 16:35:51.420723 24825 webserver.cc:460] Webserver started at http://127.17.205.61:44723/ using document root <none> and password file <none>
I20260929 16:35:51.421394 24825 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:51.421476 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:51.421728 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:51.424773 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/instance:
uuid: "78a4809edc9f464daa2f62bee4d3ad47"
format_stamp: "Formatted at 2026-09-29 16:35:51 on dist-test-slave-3h01"
server_key: "8a181537dcdb5209b958b1c9827eb1fc"
server_key_iv: "718b277d13541fb5258d7923b2dc59c4"
server_key_version: "encryptionkey@0"
I20260929 16:35:51.425552 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal/instance:
uuid: "78a4809edc9f464daa2f62bee4d3ad47"
format_stamp: "Formatted at 2026-09-29 16:35:51 on dist-test-slave-3h01"
server_key: "8a181537dcdb5209b958b1c9827eb1fc"
server_key_iv: "718b277d13541fb5258d7923b2dc59c4"
server_key_version: "encryptionkey@0"
I20260929 16:35:51.430778 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:35:51.434171 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:51.435654 24825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:51.435811 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "78a4809edc9f464daa2f62bee4d3ad47"
format_stamp: "Formatted at 2026-09-29 16:35:51 on dist-test-slave-3h01"
server_key: "8a181537dcdb5209b958b1c9827eb1fc"
server_key_iv: "718b277d13541fb5258d7923b2dc59c4"
server_key_version: "encryptionkey@0"
I20260929 16:35:51.436028 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:51.453074 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:51.456681 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:51.456898 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:51.472986 24825 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.61:38359
I20260929 16:35:51.473034 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.61:38359 every 8 connection(s)
I20260929 16:35:51.474304 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
I20260929 16:35:51.477924 24892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:51.479581 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24825
I20260929 16:35:51.487136 24892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699751, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.61@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:51.504141 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.481804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.17.205.61:38359)
0929 16:35:51.482619 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.482634 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.482709 (+ 75us) server_negotiation.cc:407] Connection header received
0929 16:35:51.484382 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.484413 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.484939 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.485388 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.486792 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.488707 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.490277 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.491286 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.491462 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.495632 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.495641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.495653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.495692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.498890 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.499662 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.499670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.499678 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.499830 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.500422 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.500428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.500431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.501136 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.501437 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.502404 (+ 967us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.502814 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"spinlock_wait_cycles":22784,"thread_start_us":344,"threads_started":1}
I20260929 16:35:51.504827 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.489951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.17.205.62:46139)
0929 16:35:51.490099 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.490105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.490748 (+ 643us) server_negotiation.cc:407] Connection header received
0929 16:35:51.490924 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.490928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.490991 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.491059 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.492423 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.493514 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.494773 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.495243 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.495368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.498929 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.498933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.498937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.498969 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.501517 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.502392 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.502398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.502401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.502560 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.503040 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.503044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.503048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.503489 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.503619 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.504181 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.504620 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:35:51.505450 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42320)
0929 16:35:51.490599 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.490640 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.490797 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.491146 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.491153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.492009 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.492270 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.492279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.493645 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.493649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.494636 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.494647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.495491 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.495499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.495651 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.496264 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.496289 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.498757 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.501668 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.501678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.501685 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.502265 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.502700 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.502705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.502710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.502880 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.503640 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.503648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.503838 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.504064 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.505176 (+ 1112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260929 16:35:51.507230 24892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699751, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.61@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
I20260929 16:35:51.518237 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.507979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:40984)
0929 16:35:51.508149 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.508166 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.508246 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.508500 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.508503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.508779 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.508906 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.508913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.509858 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.509862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.510687 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.510697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.511386 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.511390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.511570 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.512075 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.512090 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.513910 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.516381 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.516385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.516387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.516725 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.517007 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.517010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.517012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.517123 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.517568 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.517571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.517694 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.517794 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.517991 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":16}
I20260929 16:35:51.518328 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.507967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.17.205.61:38359)
0929 16:35:51.508123 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.508127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.508206 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:35:51.508329 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.508332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.508390 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.508471 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.509042 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.509735 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.510807 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.511277 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.511374 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.514072 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.514077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.514080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.514106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.516268 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.516812 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.516816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.516818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.516922 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.517204 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.517207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.517209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.517483 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.517582 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.517863 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.518089 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":48}
I20260929 16:35:51.522781 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Bootstrap starting.
I20260929 16:35:51.525906 24892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:51.526949 24892 log.cc:824] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:51.529592 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: No bootstrap required, opened a new log
I20260929 16:35:51.531644 24687 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
I20260929 16:35:51.536878 24892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } }
I20260929 16:35:51.537199 24892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:51.537287 24892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a4809edc9f464daa2f62bee4d3ad47, State: Initialized, Role: FOLLOWER
I20260929 16:35:51.537875 24892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } }
I20260929 16:35:51.538656 24898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } } }
I20260929 16:35:51.538951 24898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:51.539743 24892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:35:51.547245 24909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:51.547369 24909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:51.549736 24825 master_runner.cc:428] Detected that this master 78a4809edc9f464daa2f62bee4d3ad47 is joining an existing cluster
I20260929 16:35:51.549813 24825 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.205.61:38359
I20260929 16:35:51.569059 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.553204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42322)
0929 16:35:51.554057 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.554075 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.554271 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.554936 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.554941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.555618 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.555974 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.555987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.557220 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.557224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.558546 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.558562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.559972 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.559979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.560211 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.560947 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.560973 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.562196 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.565806 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.565811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.565815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.566282 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.566755 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.566761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.566764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.566977 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.567737 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.567743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.567996 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.568186 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.568523 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":1,"spinlock_wait_cycles":5504,"thread_start_us":956,"threads_started":1}
I20260929 16:35:51.570047 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.553142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.17.205.62:46139)
0929 16:35:51.553310 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.553315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.554152 (+ 837us) server_negotiation.cc:407] Connection header received
0929 16:35:51.554454 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.554459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.554538 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.554624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.556120 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.557095 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.558702 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.559501 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.559614 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.562405 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.562412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.562417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.562457 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.565645 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.566431 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.566437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.566441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.566601 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.567093 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.567099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.567103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.567513 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.567663 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.569425 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.569663 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:51.571025 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.553610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.17.205.61:38359)
0929 16:35:51.553765 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.553770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.554505 (+ 735us) server_negotiation.cc:407] Connection header received
0929 16:35:51.554638 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.554643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.554705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.554792 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.555794 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.556829 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.559717 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.560489 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.560924 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.563646 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.563652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.563657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.563688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.567309 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.568090 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.568095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.568098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.568237 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.568628 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.568632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.568635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.569005 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.570378 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.570518 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.570700 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":55168}
I20260929 16:35:51.571566 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.553337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:40990)
0929 16:35:51.554411 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.554427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.554538 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.554829 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.554832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.555376 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.555670 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.555678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.557020 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.557025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.559580 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.559594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.561245 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.561251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.561557 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.562158 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.562179 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.563482 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.567449 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.567453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.567457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.567953 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.568351 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.568355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.568359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.568512 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.569453 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.569458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.569600 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.569733 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.570009 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":25,"spinlock_wait_cycles":1920,"thread_start_us":444,"threads_started":1}
WARNING: no policy specified for oryx/127.17.205.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.60@KRBTEST.COM" created.
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699749, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.62@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
I20260929 16:35:51.592312 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.576820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.17.205.61:38359)
0929 16:35:51.576999 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.577004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.577623 (+ 619us) server_negotiation.cc:407] Connection header received
0929 16:35:51.578004 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.578010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.578106 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.578216 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.579609 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.580735 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.582691 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.583486 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.583594 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.587134 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.587139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.587142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.587176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.589697 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.590337 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.590343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.590347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.590482 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.590831 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.590835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.590837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.591094 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.591252 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.591745 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.591957 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:35:51.592352 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.576688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:41002)
0929 16:35:51.577492 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.577539 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.577758 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.578273 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.578281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.579111 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.579413 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.579426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.580868 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.580873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.582514 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.582529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.583646 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.583653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.583866 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.584508 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.584532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.586952 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.589823 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.589836 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.589844 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.590230 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.590588 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.590594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.590598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.590731 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.591261 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.591270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.591447 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.591629 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.591952 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":421,"threads_started":1}
I20260929 16:35:51.594391 24364 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.205.61:38359: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47"
member_type: NON_VOTER
last_known_addr {
host: "127.17.205.61"
port: 38359
}
attrs {
promote: true
}
}
dest_uuid: "6f6e48515165498ea244e5bc2e35e080"
cas_config_opid_index: -1
I20260929 16:35:51.595360 24364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } }
I20260929 16:35:51.610431 24922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } }
I20260929 16:35:51.611454 24867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:51.614984 24922 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.205.61:38359
I20260929 16:35:51.615432 24921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } }
I20260929 16:35:51.615760 24921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:51.616279 24825 master.cc:537] Master@127.17.205.61:38359 shutting down...
I20260929 16:35:51.617511 24867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:35:51.618641 24921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:35:51.622076 24348 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.60.keytab.
Entry for principal oryx/127.17.205.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.60.keytab.
I20260929 16:35:51.624882 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:40579
--webserver_interface=127.17.205.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.60.keytab
--principal=oryx/127.17.205.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:51.625422 24348 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 -> Peer 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359): Couldn't send request to peer 78a4809edc9f464daa2f62bee4d3ad47. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:35:51.630699 24825 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:51.631175 24825 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:51.631402 24825 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: stopping tablet replica
I20260929 16:35:51.638441 24825 master.cc:552] Master@127.17.205.61:38359 shutdown complete.
I20260929 16:35:51.638583 24825 master_runner.cc:305] Clearing existing system tablet
I20260929 16:35:51.639048 24825 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:51.639191 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:51.642642 24825 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 16:35:51.644384 24927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:51.644749 24825 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:51.645329 24825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 16:35:51.645503 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "78a4809edc9f464daa2f62bee4d3ad47"
format_stamp: "Formatted at 2026-09-29 16:35:51 on dist-test-slave-3h01"
server_key: "8a181537dcdb5209b958b1c9827eb1fc"
server_key_iv: "718b277d13541fb5258d7923b2dc59c4"
server_key_version: "encryptionkey@0"
I20260929 16:35:51.651767 24825 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:51.658872 24825 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:35:51.659160 24825 log.cc:1196] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:35:51.659669 24825 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Deleting consensus metadata
I20260929 16:35:51.659991 24825 master_runner.cc:315] Copying system tablet from 127.17.205.62:46139
I20260929 16:35:51.662425 24825 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.205.62:46139
I20260929 16:35:51.677567 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.663676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.17.205.62:46139)
0929 16:35:51.664714 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:51.664720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:51.664740 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:35:51.664910 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:51.664916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:51.664998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:51.665077 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:51.666145 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.667262 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.669146 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.669743 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.669845 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.672312 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:51.672318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:51.672323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:51.672357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:51.674880 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.675593 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.675598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.675602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.675719 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:51.676096 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:51.676101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:51.676104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:51.676437 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:51.676869 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:51.676992 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:35:51.677199 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919}
I20260929 16:35:51.677604 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:51.663295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42334)
0929 16:35:51.664606 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:51.664631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:51.664784 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:51.665123 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:51.665128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:51.665663 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:51.665951 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.665964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.667472 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.667478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.668969 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:51.668984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:51.670118 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:51.670124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:51.670298 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:51.671049 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:51.671072 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:51.672084 (+ 1012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:51.674998 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.675004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.675013 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.675492 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.675839 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:51.675842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:51.675845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:51.675983 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:51.676545 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:51.676551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:51.676733 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 16:35:51.676857 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:35:51.677198 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":830,"threads_started":1}
I20260929 16:35:51.678923 24384 tablet_copy_service.cc:140] P 6f6e48515165498ea244e5bc2e35e080: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 78a4809edc9f464daa2f62bee4d3ad47 ({username='oryx', principal='oryx/127.17.205.61@KRBTEST.COM'} at 127.0.0.1:42334)
I20260929 16:35:51.679194 24384 tablet_copy_service.cc:161] P 6f6e48515165498ea244e5bc2e35e080: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 78a4809edc9f464daa2f62bee4d3ad47 at {username='oryx', principal='oryx/127.17.205.61@KRBTEST.COM'} at 127.0.0.1:42334: session id = 78a4809edc9f464daa2f62bee4d3ad47-00000000000000000000000000000000
I20260929 16:35:51.682350 24384 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:35:51.684216 24825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:35:51.688688 24825 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: tablet copy: Starting download of 0 data blocks...
I20260929 16:35:51.688951 24825 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: tablet copy: Starting download of 1 WAL segments...
I20260929 16:35:51.691170 24825 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:35:51.694409 24384 tablet_copy_service.cc:342] P 6f6e48515165498ea244e5bc2e35e080: Request end of tablet copy session 78a4809edc9f464daa2f62bee4d3ad47-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.205.61@KRBTEST.COM'} at 127.0.0.1:42334
I20260929 16:35:51.694620 24384 tablet_copy_service.cc:433] P 6f6e48515165498ea244e5bc2e35e080: ending tablet copy session 78a4809edc9f464daa2f62bee4d3ad47-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 78a4809edc9f464daa2f62bee4d3ad47
I20260929 16:35:51.705716 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:51.708397 24934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:51.708778 24935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:51.716317 24825 server_base.cc:1034] running on GCE node
W20260929 16:35:51.716631 24937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:51.717067 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:51.722780 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:51.724025 24825 hybrid_clock.cc:648] HybridClock initialized: now 1790699751723930 us; error 57 us; skew 500 ppm
I20260929 16:35:51.724274 24825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:51 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699751, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:51.726387 24825 init.cc:377] Logged in from keytab as oryx/127.17.205.61@KRBTEST.COM (short username oryx)
I20260929 16:35:51.728168 24825 webserver.cc:460] Webserver started at http://127.17.205.61:34625/ using document root <none> and password file <none>
I20260929 16:35:51.728523 24825 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:51.728602 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:51.731578 24825 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:51.733399 24944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:51.733803 24825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 16:35:51.733942 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "78a4809edc9f464daa2f62bee4d3ad47"
format_stamp: "Formatted at 2026-09-29 16:35:51 on dist-test-slave-3h01"
server_key: "8a181537dcdb5209b958b1c9827eb1fc"
server_key_iv: "718b277d13541fb5258d7923b2dc59c4"
server_key_version: "encryptionkey@0"
I20260929 16:35:51.734100 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:51.761480 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:51.765280 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:51.786396 24825 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.61:38359
I20260929 16:35:51.786441 24995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.61:38359 every 8 connection(s)
I20260929 16:35:51.787484 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
I20260929 16:35:51.792441 24996 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:35:51.796218 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Bootstrap starting.
I20260929 16:35:51.813666 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:35:51.814267 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: Bootstrap complete.
I20260929 16:35:51.815399 24996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } }
I20260929 16:35:51.815685 24996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 78a4809edc9f464daa2f62bee4d3ad47, State: Initialized, Role: LEARNER
I20260929 16:35:51.815907 24996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } }
I20260929 16:35:51.816807 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } }
I20260929 16:35:51.817075 24996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:51.817260 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:35:51.824378 25010 catalog_manager.cc:1259] Loaded cluster ID: a879cd01851c48db81d9e05bc6417677
I20260929 16:35:51.824486 25010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: loading cluster ID for follower catalog manager: success
I20260929 16:35:51.827732 25010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: acquiring CA information for follower catalog manager: success
I20260929 16:35:51.829919 25010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:35:51.995560 24925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:51.995864 24925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:51.995951 24925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:35:51.998226 24820 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
W20260929 16:35:52.003605 24925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:52.003723 24925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:52.003798 24925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:52.003875 24925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:52.003931 24925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:52.014398 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.008139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:41010)
0929 16:35:52.008297 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.008313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.008415 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.009407 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.009413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.009446 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:52.009647 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.009658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.011009 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.011015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.012328 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.012344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.013558 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.013564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.013771 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.013840 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.013932 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.014077 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 16:35:52.014334 24925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.60.keytab
--principal=oryx/127.17.205.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.205.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:52.014441 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.008411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.17.205.61:38359)
0929 16:35:52.009012 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.009017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.009040 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:35:52.009112 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.009116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.009189 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.009320 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:52.009799 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.010844 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.012500 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.013379 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.013510 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.013779 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.013982 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.014116 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":314,"threads_started":1}
I20260929 16:35:52.015658 24971 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 16:35:52.015889 24925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:52.016229 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6e48515165498ea244e5bc2e35e080. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } }
I20260929 16:35:52.016469 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:35:52.017385 24925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:52.028633 25017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.028676 25019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.028600 25016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:52.029515 24925 server_base.cc:1034] running on GCE node
I20260929 16:35:52.030288 24925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:52.032094 24925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:52.033311 24925 hybrid_clock.cc:648] HybridClock initialized: now 1790699752033260 us; error 48 us; skew 500 ppm
I20260929 16:35:52.033635 24925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:52.036808 24925 init.cc:377] Logged in from keytab as oryx/127.17.205.60@KRBTEST.COM (short username oryx)
I20260929 16:35:52.038201 24925 webserver.cc:460] Webserver started at http://127.17.205.60:41231/ using document root <none> and password file <none>
I20260929 16:35:52.038851 24925 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:52.038954 24925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:52.039199 24925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:52.042234 24925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/instance:
uuid: "40511ab026ed43258598a1f16d3ad32b"
format_stamp: "Formatted at 2026-09-29 16:35:52 on dist-test-slave-3h01"
server_key: "e2629c4e38d74a71d92dae0d69b7946f"
server_key_iv: "34663ab300ba90a9a8d24e1ffffe7a67"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.042951 24925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal/instance:
uuid: "40511ab026ed43258598a1f16d3ad32b"
format_stamp: "Formatted at 2026-09-29 16:35:52 on dist-test-slave-3h01"
server_key: "e2629c4e38d74a71d92dae0d69b7946f"
server_key_iv: "34663ab300ba90a9a8d24e1ffffe7a67"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.048005 24925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:35:52.051379 25026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.052831 24925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:52.052979 24925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "40511ab026ed43258598a1f16d3ad32b"
format_stamp: "Formatted at 2026-09-29 16:35:52 on dist-test-slave-3h01"
server_key: "e2629c4e38d74a71d92dae0d69b7946f"
server_key_iv: "34663ab300ba90a9a8d24e1ffffe7a67"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.053146 24925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:52.073498 24925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:52.077610 24925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:52.077836 24925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:52.094004 24925 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.60:40579
I20260929 16:35:52.094026 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.60:40579 every 8 connection(s)
I20260929 16:35:52.095525 24925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
I20260929 16:35:52.099336 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:52.101642 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 24925
I20260929 16:35:52.109066 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.60@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:52.126539 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.103824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.17.205.60:40579)
0929 16:35:52.104599 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.104616 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.104679 (+ 63us) server_negotiation.cc:407] Connection header received
0929 16:35:52.106373 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.106408 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.106966 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.107439 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.108707 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.110726 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.112255 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.113335 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.113503 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.118405 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.118417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.118439 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.118498 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.122074 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.122920 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.122926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.122934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.123081 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.123491 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.123495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.123497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.124026 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.124301 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.124802 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.125273 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":301,"threads_started":1}
I20260929 16:35:52.127724 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.112048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42342)
0929 16:35:52.112922 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.112968 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.113135 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.113480 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.113487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.114283 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.114557 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.114567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.116049 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.116054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.117457 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.117475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.118450 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.118456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.118666 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.119360 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.119401 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.122081 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.125243 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.125255 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.125262 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.125686 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.126067 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.126071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.126073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.126195 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.126834 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.126841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.127015 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.127178 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.127459 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":425,"threads_started":1}
I20260929 16:35:52.127871 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.112224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.17.205.62:46139)
0929 16:35:52.112435 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.112441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.113067 (+ 626us) server_negotiation.cc:407] Connection header received
0929 16:35:52.113245 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.113248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.113311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.113384 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.114709 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.115903 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.117638 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.118303 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.118393 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.122285 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.122291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.122295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.122339 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.125092 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.125806 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.125811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.125814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.125963 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.126282 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.126287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.126290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.126616 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.126757 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.127319 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.127553 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:35:52.129117 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.60@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
I20260929 16:35:52.140971 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.129809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:41024)
0929 16:35:52.130008 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.130027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.130118 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.130395 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.130400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.130699 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.130830 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.130838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.131847 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.131851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.132729 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.132740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.133410 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.133415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.133587 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.134090 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.134107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.136078 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.138793 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.138798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.138802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.139148 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.139503 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.139508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.139512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.139629 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.140084 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.140089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.140221 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.140328 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.140642 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:35:52.140980 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.129922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.17.205.61:38359)
0929 16:35:52.130090 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.130096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.130112 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:35:52.130221 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.130227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.130292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.130367 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.130967 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.131718 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.132852 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.133271 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.133378 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.136274 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.136280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.136284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.136316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.138667 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.139267 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.139273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.139277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.139404 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.139720 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.139725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.139729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.139981 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.140104 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.140440 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.140657 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:35:52.142024 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.60@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
I20260929 16:35:52.152624 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.142604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45658)
0929 16:35:52.142798 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.142812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.142885 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.143098 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.143101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.143442 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.143584 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.143592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.144413 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.144416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.145224 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.145234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.145834 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.145838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.145993 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.146490 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.146507 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.148248 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.150683 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.150687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.150690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.151006 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.151307 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.151310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.151312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.151442 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.151940 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.151943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.152064 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.152162 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.152340 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:35:52.152695 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.142731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.17.205.60:40579)
0929 16:35:52.142856 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.142860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.142873 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:35:52.142953 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.142956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.143004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.143077 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.143688 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.144320 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.145329 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.145710 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.145797 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.148403 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.148408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.148411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.148437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.150578 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.151093 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.151097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.151099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.151195 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.151564 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.151568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.151570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.151839 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.151957 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.152239 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.152453 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":42}
I20260929 16:35:52.157227 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Bootstrap starting.
I20260929 16:35:52.160357 25078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:52.161306 25078 log.cc:824] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:52.163780 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: No bootstrap required, opened a new log
I20260929 16:35:52.170404 25078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } }
I20260929 16:35:52.170706 25078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:52.170794 25078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40511ab026ed43258598a1f16d3ad32b, State: Initialized, Role: FOLLOWER
I20260929 16:35:52.171372 25078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } }
I20260929 16:35:52.172174 25084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } } }
I20260929 16:35:52.172462 25084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:52.173138 25078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:52.181041 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24425
W20260929 16:35:52.182775 25095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:52.182924 25095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:52.186110 24925 master_runner.cc:428] Detected that this master 40511ab026ed43258598a1f16d3ad32b is joining an existing cluster
I20260929 16:35:52.186245 24925 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.205.60:40579
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.1@KRBTEST.COM
I20260929 16:35:52.204309 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.182716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.17.205.1:35643)
0929 16:35:52.183416 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.183425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.183453 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:35:52.183520 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.183529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.183793 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.184018 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.185610 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.187121 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.188994 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.189891 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.190616 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.195682 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.195698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.195716 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.195758 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.199495 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.200410 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.200419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.200422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.200580 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.201476 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.201480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.201483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.201914 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.203459 (+ 1545us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.203628 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.203903 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":361,"threads_started":1}
I20260929 16:35:52.205826 24548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:52.210007 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.194080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.17.205.62:46139)
0929 16:35:52.194276 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.194282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.195556 (+ 1274us) server_negotiation.cc:407] Connection header received
0929 16:35:52.195804 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.195809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.195890 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.195960 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.197395 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.198525 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.200521 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.201166 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.201258 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.203737 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.203743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.203747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.203784 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.206814 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.207625 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.207630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.207634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.207782 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.208257 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.208262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.208265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.208600 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.208737 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.209404 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.209638 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:35:52.210999 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.194100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42348)
0929 16:35:52.195463 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.195481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.195665 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.196166 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.196171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.196823 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.197209 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.197222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.198703 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.198709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.200351 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.200379 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.201546 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.201553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.201731 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.202409 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.202433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.203543 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.206969 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.206975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.206983 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.207485 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.207924 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.207929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.207933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.208122 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.208817 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.208826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.209058 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.210241 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.210564 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":12059,"threads_started":1}
I20260929 16:35:52.252913 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.206693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.17.205.60:40579)
0929 16:35:52.206875 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.206881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.211310 (+ 4429us) server_negotiation.cc:407] Connection header received
0929 16:35:52.211510 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.211515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.211584 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.211648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.212643 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.213661 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.216090 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.216840 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.216961 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.246127 (+ 29166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.246137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.246143 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.246191 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.249941 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.250770 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.250778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.250782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.250923 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.251336 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.251340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.251344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.251716 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.251845 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.252329 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.252561 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:52.259526 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.206989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45670)
0929 16:35:52.211237 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.211256 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.211393 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.211830 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.211834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.212282 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.212514 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.212525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.214231 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.214236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.215940 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.215956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.217129 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.217135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.217286 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.218302 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.218325 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.245816 (+ 27491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.250131 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.250136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.250140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.250647 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.251044 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.251049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.251055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.251204 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.251908 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.251912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.252056 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.253151 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.257934 (+ 4783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4132,"thread_start_us":12373,"threads_started":1}
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699749, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.62@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
I20260929 16:35:52.269562 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.194650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.17.205.61:38359)
0929 16:35:52.194804 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.194810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.206141 (+ 11331us) server_negotiation.cc:407] Connection header received
0929 16:35:52.207495 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.207503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.207594 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.207666 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.210406 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.211574 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.221517 (+ 9943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.222156 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.222265 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.259366 (+ 37101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.259374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.259379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.259429 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.263640 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.266838 (+ 3198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.266845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.266849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.267020 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.267551 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.267557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.267561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.267929 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.268055 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.268941 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.269206 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:35:52.270630 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.194755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:41032)
0929 16:35:52.206012 (+ 11257us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.206035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.207237 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.207897 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.207902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.208601 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.208962 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.208974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.211799 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.211804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.221320 (+ 9516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.221336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.228587 (+ 7251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.228595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.228809 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.256766 (+ 27957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.256787 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.259100 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.263803 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.263808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.263813 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.266693 (+ 2880us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.267179 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.267184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.267188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.267402 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.268407 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.268413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.268641 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.269838 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.270187 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11110,"thread_start_us":330,"threads_started":1}
I20260929 16:35:52.275432 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.252461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45680)
0929 16:35:52.252801 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.252819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.252916 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.253669 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.253675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.254100 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.254309 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.254322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.255931 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.255936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.257337 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.257352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.258562 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.258568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.258755 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.259338 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.259381 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.262442 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.272232 (+ 9790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.272238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.272243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.272850 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.273283 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.273288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.273292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.273465 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.274304 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.274310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.274485 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.274695 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.274982 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260929 16:35:52.276553 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.253179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.17.205.60:40579)
0929 16:35:52.253300 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.253306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.253320 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:52.253425 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.253430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.253507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.254466 (+ 959us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.254623 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.255778 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.257561 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.258386 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.262601 (+ 4215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.262800 (+ 199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.262806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.262811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.262838 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.266309 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.273016 (+ 6707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.273022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.273026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.273158 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.273675 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.273679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.273683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.274080 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.275846 (+ 1766us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.275970 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.276199 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:35:52.278035 24364 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.205.60:40579: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "40511ab026ed43258598a1f16d3ad32b"
member_type: NON_VOTER
last_known_addr {
host: "127.17.205.60"
port: 40579
}
attrs {
promote: true
}
}
dest_uuid: "6f6e48515165498ea244e5bc2e35e080"
cas_config_opid_index: 5
I20260929 16:35:52.279066 24364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } }
I20260929 16:35:52.281452 24921 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 40511ab026ed43258598a1f16d3ad32b (127.17.205.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.283967 24971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:35:52.285569 25053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:52.289126 24921 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.205.60:40579
I20260929 16:35:52.289809 24925 master.cc:537] Master@127.17.205.60:40579 shutting down...
I20260929 16:35:52.294835 25053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:35:52.306203 25106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:35:52.307256 24921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.308883 24921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:52.309193 24925 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:52.309633 24925 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:52.309847 24925 tablet_replica.cc:330] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: stopping tablet replica
I20260929 16:35:52.313769 25107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:52.319188 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.318720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45690)
0929 16:35:52.318884 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.319008 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:35:52.319494 24349 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 -> Peer 40511ab026ed43258598a1f16d3ad32b (127.17.205.60:40579): Couldn't send request to peer 40511ab026ed43258598a1f16d3ad32b. Status: Network error: Client connection negotiation failed: client connection to 127.17.205.60:40579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:35:52.320168 24971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 40511ab026ed43258598a1f16d3ad32b (127.17.205.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.324565 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: NON_VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: true } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.324877 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:35:52.325791 25106 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: attempting to promote NON_VOTER 78a4809edc9f464daa2f62bee4d3ad47 to VOTER
I20260929 16:35:52.326711 25106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } }
I20260929 16:35:52.329491 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45700)
0929 16:35:52.329227 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.329320 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 16:35:52.330686 24970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 LEARNER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 16:35:52.331292 24921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 16:35:52.333390 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.332952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45706)
0929 16:35:52.333160 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.333243 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 16:35:52.333575 24349 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 -> Peer 40511ab026ed43258598a1f16d3ad32b (127.17.205.60:40579): Couldn't send request to peer 40511ab026ed43258598a1f16d3ad32b. Status: Network error: Client connection negotiation failed: client connection to 127.17.205.60:40579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:35:52.334878 24921 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.339141 25107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.339473 25107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:52.339926 25108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.340258 25108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:52.341204 24970 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.345181 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.345458 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:52.348431 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24425
I20260929 16:35:52.370146 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:35643
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:46139
--webserver_port=34879
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:35:52.707760 24925 master.cc:552] Master@127.17.205.60:40579 shutdown complete.
I20260929 16:35:52.707868 24925 master_runner.cc:305] Clearing existing system tablet
I20260929 16:35:52.708249 24925 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:52.708361 24925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:52.711181 24925 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:52.712672 25114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.712983 24925 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:52.713227 24925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:35:52.713322 24925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "40511ab026ed43258598a1f16d3ad32b"
format_stamp: "Formatted at 2026-09-29 16:35:52 on dist-test-slave-3h01"
server_key: "e2629c4e38d74a71d92dae0d69b7946f"
server_key_iv: "34663ab300ba90a9a8d24e1ffffe7a67"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.718009 24925 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:52.724782 24925 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:35:52.724941 24925 log.cc:1196] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:35:52.725383 24925 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Deleting consensus metadata
I20260929 16:35:52.725637 24925 master_runner.cc:315] Copying system tablet from 127.17.205.62:46139
I20260929 16:35:52.728073 24925 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.205.62:46139
I20260929 16:35:52.741354 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.728761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:42358)
0929 16:35:52.729579 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.729603 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.729756 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.730097 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.730102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.730590 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.730847 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.730858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.731911 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.731917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.733363 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.733377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.734065 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.734072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.734242 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.734843 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.734860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.735602 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.739232 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.739238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.739242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.739596 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.739947 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.739950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.739953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.740066 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.740637 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.740639 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.740784 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.740891 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.741090 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":541,"threads_started":1}
I20260929 16:35:52.741600 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.728848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.17.205.62:46139)
0929 16:35:52.729369 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.729376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.729683 (+ 307us) server_negotiation.cc:407] Connection header received
0929 16:35:52.729878 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.729882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.729956 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.730047 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.730973 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.731784 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.733519 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.733911 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.734003 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.735875 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.735883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.735888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.735927 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.739070 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.739706 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.739712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.739716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.739857 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.740156 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.740160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.740164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.740504 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.740690 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.741016 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.741233 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260929 16:35:52.742077 24384 tablet_copy_service.cc:140] P 6f6e48515165498ea244e5bc2e35e080: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 40511ab026ed43258598a1f16d3ad32b ({username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:42358)
I20260929 16:35:52.742236 24384 tablet_copy_service.cc:161] P 6f6e48515165498ea244e5bc2e35e080: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 40511ab026ed43258598a1f16d3ad32b at {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:42358: session id = 40511ab026ed43258598a1f16d3ad32b-00000000000000000000000000000000
I20260929 16:35:52.744527 24384 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:35:52.746238 24925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:35:52.752825 24925 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: tablet copy: Starting download of 0 data blocks...
I20260929 16:35:52.753129 24925 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: tablet copy: Starting download of 1 WAL segments...
I20260929 16:35:52.755373 24925 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:35:52.759634 24384 tablet_copy_service.cc:342] P 6f6e48515165498ea244e5bc2e35e080: Request end of tablet copy session 40511ab026ed43258598a1f16d3ad32b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:42358
I20260929 16:35:52.759753 24384 tablet_copy_service.cc:433] P 6f6e48515165498ea244e5bc2e35e080: ending tablet copy session 40511ab026ed43258598a1f16d3ad32b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 40511ab026ed43258598a1f16d3ad32b
I20260929 16:35:52.768586 24925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:52.771185 25123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.771555 25122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.772696 25125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:52.772696 24925 server_base.cc:1034] running on GCE node
I20260929 16:35:52.773090 24925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:52.773638 24925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:52.774821 24925 hybrid_clock.cc:648] HybridClock initialized: now 1790699752774785 us; error 33 us; skew 500 ppm
I20260929 16:35:52.775002 24925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:52.776644 24925 init.cc:377] Logged in from keytab as oryx/127.17.205.60@KRBTEST.COM (short username oryx)
I20260929 16:35:52.777697 24925 webserver.cc:460] Webserver started at http://127.17.205.60:37715/ using document root <none> and password file <none>
I20260929 16:35:52.777966 24925 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:52.778054 24925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:52.780164 24925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 16:35:52.781574 25132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.781946 24925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:35:52.782048 24925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "40511ab026ed43258598a1f16d3ad32b"
format_stamp: "Formatted at 2026-09-29 16:35:52 on dist-test-slave-3h01"
server_key: "e2629c4e38d74a71d92dae0d69b7946f"
server_key_iv: "34663ab300ba90a9a8d24e1ffffe7a67"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.782198 24925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 16:35:52.804168 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:52.804466 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:52.804610 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:35:52.807936 24925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:52.810794 24925 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 16:35:52.812542 25112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:52.812659 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:52.812801 25112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:35:52.824599 25112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:35643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=34879
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:52.825982 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:52.827363 25112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:35:52.828883 24925 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.60:40579
I20260929 16:35:52.828899 25186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.60:40579 every 8 connection(s)
I20260929 16:35:52.829655 24925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
I20260929 16:35:52.833073 25187 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:35:52.835609 25187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Bootstrap starting.
W20260929 16:35:52.842752 25193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.842854 25191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:52.842978 25190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:52.844480 25112 server_base.cc:1034] running on GCE node
I20260929 16:35:52.845069 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:52.846084 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:52.847294 25112 hybrid_clock.cc:648] HybridClock initialized: now 1790699752847249 us; error 40 us; skew 500 ppm
I20260929 16:35:52.847597 25112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:52.850468 25112 init.cc:377] Logged in from keytab as oryx/127.17.205.1@KRBTEST.COM (short username oryx)
I20260929 16:35:52.851395 25187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:35:52.851761 25187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Bootstrap complete.
I20260929 16:35:52.852012 25112 webserver.cc:460] Webserver started at http://127.17.205.1:34879/ using document root <none> and password file <none>
I20260929 16:35:52.852528 25187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } }
I20260929 16:35:52.852667 25112 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:52.852728 25187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40511ab026ed43258598a1f16d3ad32b, State: Initialized, Role: LEARNER
I20260929 16:35:52.852763 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:52.852926 25187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } }
I20260929 16:35:52.853550 25200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.853857 25187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:52.853881 25200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: LEARNER
I20260929 16:35:52.858680 25112 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:35:52.860175 25212 catalog_manager.cc:1259] Loaded cluster ID: a879cd01851c48db81d9e05bc6417677
I20260929 16:35:52.860262 25212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: loading cluster ID for follower catalog manager: success
I20260929 16:35:52.862926 25212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: acquiring CA information for follower catalog manager: success
I20260929 16:35:52.864624 25215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.865096 25212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:35:52.866120 25112 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260929 16:35:52.866282 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e"
format_stamp: "Formatted at 2026-09-29 16:35:49 on dist-test-slave-3h01"
server_key: "521cae2391a2cd1ac619549d88d34e4a"
server_key_iv: "bcb90da5b06a60c38eb096c54f1cefe9"
server_key_version: "encryptionkey@0"
I20260929 16:35:52.867264 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:52.868968 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.860719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.17.205.60:40579)
0929 16:35:52.863679 (+ 2960us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.863683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.863702 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:35:52.863755 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.863759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.863818 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.863920 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:52.864387 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.865689 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.867169 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.868055 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.868178 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.868364 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.868558 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.868687 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":2773,"threads_started":1}
I20260929 16:35:52.868947 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.860675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:45710)
0929 16:35:52.861152 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.861175 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.861325 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.863980 (+ 2655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.863985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.864015 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:52.864213 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.864226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.865830 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.865834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.867020 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.867035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.868229 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.868235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.868413 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.868440 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.868524 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.868671 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":782,"threads_started":1}
I20260929 16:35:52.869757 25162 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:35:52.870237 25200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6e48515165498ea244e5bc2e35e080. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:52.870496 25200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: LEARNER
I20260929 16:35:52.885480 25112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:52.889731 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:52.890015 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:52.890977 25112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:52.892366 25112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:52.892448 25112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.892532 25112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:52.892584 25112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:52.921733 25112 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:35643
I20260929 16:35:52.921792 25327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:35643 every 8 connection(s)
I20260929 16:35:52.923242 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:35:52.925812 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 25112
I20260929 16:35:52.926113 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24559
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.2@KRBTEST.COM
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
Sep 29 16:35:52 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699752, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:52.962154 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.927192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.17.205.2:45241)
0929 16:35:52.937622 (+ 10430us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.937633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.938711 (+ 1078us) server_negotiation.cc:407] Connection header received
0929 16:35:52.938800 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.938809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.939129 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.939305 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.942069 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.943562 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.945420 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946228 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.946340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.952938 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.952954 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.952974 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.953028 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.958114 (+ 5086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.958983 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.958992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.958996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.959141 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.959618 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.959623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.959627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.960051 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.960313 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.961493 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.961754 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10213,"thread_start_us":10422,"threads_started":1}
I20260929 16:35:52.962692 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.940695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:49331 (local address 127.17.205.62:46139)
0929 16:35:52.940883 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.940890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.941449 (+ 559us) server_negotiation.cc:407] Connection header received
0929 16:35:52.942797 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.942802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.942890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.942971 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.945596 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946563 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.949342 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.950032 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950128 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.955879 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.955887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.955892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.955934 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.959675 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960480 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.960485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.960488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.960645 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.961078 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.961084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.961087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.961425 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.961544 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.962266 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.962485 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:35:52.962808 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.927642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:42029 (local address 127.17.205.60:40579)
0929 16:35:52.927833 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.927839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.940015 (+ 12176us) server_negotiation.cc:407] Connection header received
0929 16:35:52.942556 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.942563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.942645 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.942715 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.945610 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946692 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.949341 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.949954 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.955892 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.955897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.955902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.955938 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.959069 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960085 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.960090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.960094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.960227 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960788 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.960793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.960797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.961104 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.961226 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.962267 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.962494 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:35:52.962852 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.926099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:57533 (local address 127.17.205.61:38359)
0929 16:35:52.926628 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:52.926635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:52.939468 (+ 12833us) server_negotiation.cc:407] Connection header received
0929 16:35:52.942561 (+ 3093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:52.942566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:52.942648 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:52.942717 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:52.945609 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946709 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.949340 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.949961 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950054 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.954888 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:52.954895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:52.954899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:52.954932 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:52.958939 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960085 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.960091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.960094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.960227 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960788 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:52.960793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:52.960797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:52.961119 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:52.961226 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:52.962261 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 16:35:52.962478 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
I20260929 16:35:52.964372 24682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:52.964498 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.927012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.17.205.1:42029)
0929 16:35:52.939909 (+ 12897us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.939926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.942230 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.943294 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.943307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.944309 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.945397 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.945435 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946884 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.946889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.949181 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.949193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.950405 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.950862 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.951827 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.952187 (+ 360us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.954926 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.959243 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.959248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.959251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.959967 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.960369 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.960378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.960663 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.961314 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.961323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.961828 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.963681 (+ 1853us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.963943 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12780,"mutex_wait_us":600,"spinlock_wait_cycles":124416,"thread_start_us":12251,"threads_started":1}
I20260929 16:35:52.964720 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.926774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.17.205.1:57533)
0929 16:35:52.939292 (+ 12518us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.939353 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.942230 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.943294 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.943307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.944328 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.945397 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.945435 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946893 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.946897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.949179 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.949193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.950405 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.950893 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.951619 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.951646 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.954647 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.959190 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.959199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.959246 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.959941 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.960369 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.960378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.960664 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.961314 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.961323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.961828 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.964026 (+ 2198us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.964290 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12131,"mutex_wait_us":495,"spinlock_wait_cycles":113408,"thread_start_us":12711,"threads_started":1}
I20260929 16:35:52.965409 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:52.940533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:49331)
0929 16:35:52.941321 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:52.941335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:52.942241 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:52.943295 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:52.943307 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:52.944329 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:52.945405 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.945437 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.946746 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.946751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.949173 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:52.949188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:52.950451 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:52.950459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:52.950824 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:52.951682 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:52.951704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:52.955662 (+ 3958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:52.959885 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.959889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.959893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.960357 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.960787 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:52.960792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:52.960795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:52.960955 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:52.961618 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:52.961622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:52.961826 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 16:35:52.962855 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0929 16:35:52.963337 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":368,"spinlock_wait_cycles":33920,"thread_start_us":534,"threads_started":1}
I20260929 16:35:52.966722 25328 heartbeater.cc:347] Connected to a master server at 127.17.205.60:40579
I20260929 16:35:52.966755 25330 heartbeater.cc:347] Connected to a master server at 127.17.205.61:38359
I20260929 16:35:52.966781 25329 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:52.967092 25330 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:52.967095 25329 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:52.967101 25328 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:52.968096 25330 heartbeater.cc:511] Master 127.17.205.61:38359 requested a full tablet report, sending...
I20260929 16:35:52.968168 25328 heartbeater.cc:511] Master 127.17.205.60:40579 requested a full tablet report, sending...
I20260929 16:35:52.969148 25329 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
I20260929 16:35:52.970264 24364 ts_manager.cc:194] Re-registered known tserver with Master: 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:52.970513 25152 ts_manager.cc:194] Registered new tserver with Master: 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:52.971405 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:49331
I20260929 16:35:52.973690 24961 ts_manager.cc:194] Registered new tserver with Master: 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:53.048038 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24559
I20260929 16:35:53.062016 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:45241
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:46139
--webserver_port=41907
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:53.386756 25339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:53.387034 25339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:53.387159 25339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:53.394322 25339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:53.394461 25339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:53.394622 25339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:35:53.395238 25106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: NON_VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: true } } }
I20260929 16:35:53.395571 25106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:53.396950 25344 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: attempting to promote NON_VOTER 40511ab026ed43258598a1f16d3ad32b to VOTER
I20260929 16:35:53.397630 25344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:53.399676 25162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 LEARNER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:35:53.399718 24970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Refusing update from remote peer 6f6e48515165498ea244e5bc2e35e080: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:35:53.400202 25344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:35:53.400580 25343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:35:53.404642 25106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40511ab026ed43258598a1f16d3ad32b (127.17.205.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.405462 25162 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40511ab026ed43258598a1f16d3ad32b (127.17.205.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.407083 25339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:45241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=41907
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:53.408180 25106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.408074 24970 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40511ab026ed43258598a1f16d3ad32b (127.17.205.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.408756 25339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:53.408860 25106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:53.409085 25106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.409272 25106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:53.408649 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.409451 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:53.410423 25339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:35:53.411859 25349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.412235 25349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:53.415591 25350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f6e48515165498ea244e5bc2e35e080" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:53.415964 25350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 16:35:53.424551 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:53.424518 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:53.424588 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:53.425232 25339 server_base.cc:1034] running on GCE node
I20260929 16:35:53.425843 25339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:53.426779 25339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:53.428000 25339 hybrid_clock.cc:648] HybridClock initialized: now 1790699753427922 us; error 62 us; skew 500 ppm
I20260929 16:35:53.428344 25339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:53.431614 25339 init.cc:377] Logged in from keytab as oryx/127.17.205.2@KRBTEST.COM (short username oryx)
I20260929 16:35:53.433082 25339 webserver.cc:460] Webserver started at http://127.17.205.2:41907/ using document root <none> and password file <none>
I20260929 16:35:53.433755 25339 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:53.433894 25339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:53.439823 25339 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:53.443181 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:53.444684 25339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:53.444847 25339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "5e8e51cda88e4922995232ba64e5172e"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "af26932385f8b5ea683e33c08a3e68d9"
server_key_iv: "68e415b4a97faf088070670c86dbafd4"
server_key_version: "encryptionkey@0"
I20260929 16:35:53.445884 25339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:53.457561 25339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:53.462026 25339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:53.462307 25339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:53.463462 25339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:53.464854 25339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:53.464948 25339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:53.465054 25339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:53.465121 25339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:53.494490 25339 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45241
I20260929 16:35:53.494563 25474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45241 every 8 connection(s)
I20260929 16:35:53.495939 25339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:35:53.501348 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 25339
I20260929 16:35:53.501627 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24692
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:53.526046 25479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.498722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:53723 (local address 127.17.205.61:38359)
0929 16:35:53.499141 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:53.499147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:53.500543 (+ 1396us) server_negotiation.cc:407] Connection header received
0929 16:35:53.503589 (+ 3046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:53.503595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:53.503670 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:53.503763 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:53.506218 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.507370 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.511009 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.511912 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.512032 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.517884 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:53.517892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:53.517897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:53.517965 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:53.521383 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.522340 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.522347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.522351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.522498 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.522967 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.522972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.522975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.523375 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:53.523526 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:53.525461 (+ 1935us) server_negotiation.cc:299] Negotiation successful
0929 16:35:53.525721 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":760,"threads_started":1}
I20260929 16:35:53.526522 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.499436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.17.205.2:53723)
0929 16:35:53.500356 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:53.500381 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:53.502774 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:53.503928 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:53.503932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:53.504900 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:53.506014 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.506039 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.507537 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.507541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.510851 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.510862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.512250 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.512256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.512443 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.515488 (+ 3045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:53.515509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:53.517675 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:53.521519 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.521530 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.521550 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.522189 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.522631 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.522637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.522641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.522851 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.523606 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:53.523614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:53.524085 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 16:35:53.524413 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 16:35:53.524970 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":1750,"spinlock_wait_cycles":86400,"thread_start_us":470,"threads_started":1}
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.3@KRBTEST.COM
I20260929 16:35:53.528294 25477 heartbeater.cc:347] Connected to a master server at 127.17.205.61:38359
I20260929 16:35:53.528638 25477 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:53.529214 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.502220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:56983 (local address 127.17.205.60:40579)
0929 16:35:53.502686 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:53.502691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:53.502711 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:35:53.503276 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:53.503281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:53.503350 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:53.503444 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:53.506209 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.507370 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.509823 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.510512 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.510626 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.519471 (+ 8845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:53.519478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:53.519483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:53.519524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:53.522554 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.523603 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.523608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.523612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.523748 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.524183 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.524187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.524190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.524513 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:53.524630 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:53.528022 (+ 3392us) server_negotiation.cc:299] Negotiation successful
0929 16:35:53.528862 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260929 16:35:53.529415 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.499436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:41487)
0929 16:35:53.500330 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:53.500382 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:53.502774 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:53.504059 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:53.504064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:53.504938 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:53.507535 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.507545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.512862 (+ 5317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.512866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.514315 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.514329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.516477 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.516488 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.516673 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.517881 (+ 1208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:53.517897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:53.522843 (+ 4946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:53.526908 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.526912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.526916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.527381 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.527865 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.527869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.527873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.528044 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.528614 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:53.528617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:53.528737 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 16:35:53.528848 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:35:53.529073 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":2284,"spinlock_wait_cycles":14848,"thread_start_us":335,"threads_started":1}
I20260929 16:35:53.529601 25477 heartbeater.cc:511] Master 127.17.205.61:38359 requested a full tablet report, sending...
I20260929 16:35:53.530079 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.498834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:41487 (local address 127.17.205.62:46139)
0929 16:35:53.499922 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:53.499928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:53.500496 (+ 568us) server_negotiation.cc:407] Connection header received
0929 16:35:53.503189 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:53.503195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:53.503299 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:53.503394 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:53.508303 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.509395 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.514450 (+ 5055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.515075 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.515198 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.523066 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:53.523073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:53.523078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:53.523118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:53.526774 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.527595 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.527602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.527605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.527767 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.528150 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.528155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.528159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.528511 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:53.528662 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:53.529525 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 16:35:53.529735 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":834,"threads_started":1}
I20260929 16:35:53.530725 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.500996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.17.205.2:56983)
0929 16:35:53.501549 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:53.501567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:53.503115 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:53.503843 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:53.503855 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:53.505060 (+ 1205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:53.506014 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.506039 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.507679 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.507683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.509697 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:53.509707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.510898 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.510905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:53.511190 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.511958 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:53.511989 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:53.519201 (+ 7212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:53.523004 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.523019 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.523022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.523472 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.523874 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:53.523878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:53.523881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:53.524059 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:53.524683 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:53.524688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:53.524816 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 16:35:53.524960 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:35:53.525201 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":250,"spinlock_wait_cycles":2432,"thread_start_us":279,"threads_started":1}
I20260929 16:35:53.530778 25476 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:53.531111 25476 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:53.531424 24961 ts_manager.cc:194] Registered new tserver with Master: 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
I20260929 16:35:53.531571 25476 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
I20260929 16:35:53.532420 24364 ts_manager.cc:194] Re-registered known tserver with Master: 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
I20260929 16:35:53.532532 25475 heartbeater.cc:347] Connected to a master server at 127.17.205.60:40579
I20260929 16:35:53.532622 25475 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:53.532904 25475 heartbeater.cc:511] Master 127.17.205.60:40579 requested a full tablet report, sending...
I20260929 16:35:53.533281 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:41487
I20260929 16:35:53.533922 25152 ts_manager.cc:194] Registered new tserver with Master: 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
I20260929 16:35:53.537405 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:53.502802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.17.205.3:32901)
0929 16:35:53.515657 (+ 12855us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:53.515664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:53.515687 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:35:53.515736 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:53.515742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:53.515947 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:53.516085 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:53.517763 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.519365 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.521249 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:53.521917 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:53.522032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:53.528068 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:53.528083 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:53.528100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:53.528148 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:53.534752 (+ 6604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.535651 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.535659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.535661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.535811 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:53.536182 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:53.536185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:53.536187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:53.536482 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:53.536678 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:53.536953 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:35:53.537139 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12718,"thread_start_us":12562,"threads_started":1}
I20260929 16:35:53.538625 24815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:53.619897 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24692
I20260929 16:35:53.632755 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:32901
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:46139
--webserver_port=38825
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:53.953699 25488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:53.953994 25488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:53.954142 25488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:53.961230 25488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:53.961347 25488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:53.961499 25488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:35:53.970942 25488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:32901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=38825
--tserver_master_addrs=127.17.205.62:46139,127.17.205.61:38359,127.17.205.60:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:35:53.972319 25488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:53.973729 25488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:35:53.974401 25329 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
W20260929 16:35:53.987227 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:53.987197 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:53.987720 25493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:53.988236 25488 server_base.cc:1034] running on GCE node
I20260929 16:35:53.988790 25488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:53.989727 25488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:53.990929 25488 hybrid_clock.cc:648] HybridClock initialized: now 1790699753990867 us; error 60 us; skew 500 ppm
I20260929 16:35:53.991261 25488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:35:53 dist-test-slave-3h01 krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:53.994257 25488 init.cc:377] Logged in from keytab as oryx/127.17.205.3@KRBTEST.COM (short username oryx)
I20260929 16:35:53.995671 25488 webserver.cc:460] Webserver started at http://127.17.205.3:38825/ using document root <none> and password file <none>
I20260929 16:35:53.996326 25488 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:53.996444 25488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:54.001792 25488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:54.004547 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:54.005926 25488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:54.006072 25488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "9ee28e215f494ab193143e44161047e8"
format_stamp: "Formatted at 2026-09-29 16:35:50 on dist-test-slave-3h01"
server_key: "dd7dfe1449c06645d63f6870e72d6f8a"
server_key_iv: "7ca05e136f86a63dfa70e2224e7541e3"
server_key_version: "encryptionkey@0"
I20260929 16:35:54.007005 25488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:54.028669 25488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:54.033202 25488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:54.033473 25488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:54.034473 25488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:35:54.035938 25488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:35:54.036029 25488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:54.036130 25488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:35:54.036203 25488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:54.065721 25488 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:32901
I20260929 16:35:54.065775 25615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:32901 every 8 connection(s)
I20260929 16:35:54.067138 25488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:35:54.071426 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 25488
Sep 29 16:35:54 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.61@KRBTEST.COM
Sep 29 16:35:54 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:35:54.102972 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.069932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45059 (local address 127.17.205.61:38359)
0929 16:35:54.076991 (+ 7059us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.076999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.077693 (+ 694us) server_negotiation.cc:407] Connection header received
0929 16:35:54.080798 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.080805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.080918 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.081054 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.084224 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.085439 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.089918 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.090569 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.090698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.095759 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.095766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.095771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.095818 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.099091 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.100172 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.100177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.100180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.100331 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.100853 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.100858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.100861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.101222 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.101359 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.102406 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.102607 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6930,"thread_start_us":6831,"threads_started":1}
I20260929 16:35:54.104580 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.070614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.17.205.3:45059)
0929 16:35:54.077486 (+ 6872us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.077547 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.080392 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.081344 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.081358 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.082780 (+ 1422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:54.083967 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.084000 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.085657 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.085663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.089772 (+ 4109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.089788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.091002 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.091010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.091367 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.092249 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:54.092282 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:54.095505 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:54.099465 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.099474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.099494 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.100059 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.100454 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.100458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.100461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.100744 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.101336 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:54.101344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:54.101835 (+ 491us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.102208 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.102757 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6498,"mutex_wait_us":2488,"thread_start_us":6250,"threads_started":1}
I20260929 16:35:54.107177 25618 heartbeater.cc:347] Connected to a master server at 127.17.205.61:38359
I20260929 16:35:54.107661 25618 heartbeater.cc:464] Registering TS with master...
Sep 29 16:35:54 dist-test-slave-3h01 krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699753, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.60@KRBTEST.COM
I20260929 16:35:54.108680 25618 heartbeater.cc:511] Master 127.17.205.61:38359 requested a full tablet report, sending...
I20260929 16:35:54.110764 24961 ts_manager.cc:194] Registered new tserver with Master: 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901)
I20260929 16:35:54.113617 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.070614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:48021)
0929 16:35:54.082166 (+ 11552us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.082191 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.082307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.083052 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.083057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.083743 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:54.084090 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.084103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.085806 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.085813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.088178 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.088199 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.095773 (+ 7574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.095780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.095948 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.096744 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:54.096765 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:54.099331 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:54.104981 (+ 5650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.104986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.104990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.105488 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.106107 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.106111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.106115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.106257 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.112600 (+ 6343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:54.112606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:54.112763 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.112899 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.113177 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11392,"mutex_wait_us":39,"spinlock_wait_cycles":73472,"thread_start_us":11086,"threads_started":1}
I20260929 16:35:54.113685 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.069965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:48021 (local address 127.17.205.62:46139)
0929 16:35:54.082694 (+ 12729us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.082700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.082721 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:35:54.082779 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.082786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.082868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.084255 (+ 1387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.084414 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.085619 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.090017 (+ 4398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.090749 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.099331 (+ 8582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.099583 (+ 252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.099590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.099595 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.099630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.104869 (+ 5239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.105628 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.105634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.105638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.105817 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.112065 (+ 6248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.112071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.112075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.112470 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.112629 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.113030 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.113297 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12617,"thread_start_us":12505,"threads_started":1}
I20260929 16:35:54.114388 25617 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46139
I20260929 16:35:54.114517 25617 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:54.114929 25617 heartbeater.cc:511] Master 127.17.205.62:46139 requested a full tablet report, sending...
I20260929 16:35:54.116271 24364 ts_manager.cc:194] Re-registered known tserver with Master: 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901)
I20260929 16:35:54.117367 24364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:48021
I20260929 16:35:54.118347 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.088652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.17.205.3:55861)
0929 16:35:54.096257 (+ 7605us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.096277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.096398 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.097377 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.097382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.098003 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:54.098316 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.098325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.100881 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.100887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.102408 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.102422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.106034 (+ 3612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.106040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.106230 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.106751 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:54.106770 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:54.109548 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:54.115602 (+ 6054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.115607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.115610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.116058 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.116486 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:54.116491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:54.116494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:54.116654 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:54.117323 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:54.117328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:54.117532 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.117678 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.117941 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7476,"thread_start_us":444,"threads_started":1}
I20260929 16:35:54.119258 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.082606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:55861 (local address 127.17.205.60:40579)
0929 16:35:54.095399 (+ 12793us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.095405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.096512 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 16:35:54.096580 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.096585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.096660 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.096745 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.098508 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.099603 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.105048 (+ 5445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.105685 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.105802 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.109762 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.109769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.109774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.111227 (+ 1453us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.115457 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.116199 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.116205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.116208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.116355 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.116781 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.116787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.116790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.117113 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.117250 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.118692 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.118905 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12556,"thread_start_us":12610,"threads_started":1}
I20260929 16:35:54.120579 25616 heartbeater.cc:347] Connected to a master server at 127.17.205.60:40579
I20260929 16:35:54.120719 25616 heartbeater.cc:464] Registering TS with master...
I20260929 16:35:54.121129 25616 heartbeater.cc:511] Master 127.17.205.60:40579 requested a full tablet report, sending...
I20260929 16:35:54.122251 25152 ts_manager.cc:194] Registered new tserver with Master: 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901)
I20260929 16:35:54.125726 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.090236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.17.205.61:38359)
0929 16:35:54.103235 (+ 12999us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.103242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.106626 (+ 3384us) server_negotiation.cc:407] Connection header received
0929 16:35:54.106899 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.106904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.106984 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.107050 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.110040 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.111260 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.114727 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.115404 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.115541 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.119397 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.119403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.119408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.119444 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.122595 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.123433 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.123439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.123442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.123586 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.124020 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.124025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.124029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.124348 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.124615 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.125179 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.125389 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12874,"thread_start_us":14358,"threads_started":1}
I20260929 16:35:54.135018 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.109062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.17.205.60:40579)
0929 16:35:54.119589 (+ 10527us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.119594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.121304 (+ 1710us) server_negotiation.cc:407] Connection header received
0929 16:35:54.122899 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.122906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.122991 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.123096 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.124529 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.125313 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.126880 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.127397 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.127502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.129846 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.129850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.129853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.129881 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.132268 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.132910 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.132915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.132918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.133040 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.133410 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.133415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.133418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.133664 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.133880 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.134292 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.134620 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10400,"thread_start_us":13909,"threads_started":1}
I20260929 16:35:54.137527 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.089979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.17.205.62:46139)
0929 16:35:54.104428 (+ 14449us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.104435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.108437 (+ 4002us) server_negotiation.cc:407] Connection header received
0929 16:35:54.108581 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.108586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.108659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.108752 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.123283 (+ 14531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.124393 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.126134 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.126618 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.126730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.129733 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.129737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.129741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.129773 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.132208 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.132909 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.132913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.132915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.133035 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.133390 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.133394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.133396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.133643 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.133860 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.134881 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.135105 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14301,"thread_start_us":14122,"threads_started":1}
I20260929 16:35:54.141862 24364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:54.148662 24364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:54.185410 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.166414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.17.205.3:32901)
0929 16:35:54.174763 (+ 8349us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.174773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.174813 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:35:54.174924 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.174932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.175225 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.175447 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:54.176345 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.178173 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.182191 (+ 4018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.184339 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.184495 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.184723 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.184946 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.185107 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8170,"thread_start_us":8016,"threads_started":1}
I20260929 16:35:54.185416 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.166157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:32901 (local address 127.0.0.1:44958)
0929 16:35:54.167755 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.167776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.167896 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.175663 (+ 7767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.175669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.175699 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:54.176125 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.176138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.178602 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.178607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.182033 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.182044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.184556 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.184563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.184747 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.184786 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.184883 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.185041 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"thread_start_us":7932,"threads_started":1}
I20260929 16:35:54.188786 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.168287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:35643 (local address 127.0.0.1:59510)
0929 16:35:54.178157 (+ 9870us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.178177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.178285 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.181227 (+ 2942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.181232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.181257 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:54.181747 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.181761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.184427 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.184432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.186003 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.186016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.187947 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.187953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.188126 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.188161 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.188247 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.188377 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9741,"mutex_wait_us":1,"thread_start_us":10053,"threads_started":1}
I20260929 16:35:54.188978 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.166830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.17.205.2:45241)
0929 16:35:54.169518 (+ 2688us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.169529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.174546 (+ 5017us) server_negotiation.cc:407] Connection header received
0929 16:35:54.174881 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.174892 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.175232 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.175418 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:54.176968 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.178464 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.186868 (+ 8404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.187992 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.188124 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.188317 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.188550 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.188718 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":2418,"threads_started":1}
I20260929 16:35:54.189015 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.166601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45241 (local address 127.0.0.1:53046)
0929 16:35:54.174392 (+ 7791us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.174414 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.174682 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.176366 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.176371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.176399 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:54.176767 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.176778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.183290 (+ 6512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.183294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.186707 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.186719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.188198 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.188204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.188370 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.188404 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.188482 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.188611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7635,"thread_start_us":864,"threads_started":1}
I20260929 16:35:54.189896 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.168540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.17.205.1:35643)
0929 16:35:54.180251 (+ 11711us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.180261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.180293 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:35:54.180422 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.180431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.180764 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.180989 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:54.181953 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.184295 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.186148 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.187461 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.187594 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.187805 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.189465 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.189626 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11535,"thread_start_us":11427,"threads_started":1}
I20260929 16:35:54.189849 25551 tablet_service.cc:1467] Processing CreateTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 (DEFAULT_TABLE table=test-table [id=9189e28cc0d345acb1e9cee68dee594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:54.191231 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6ff8ec69d747f59256fcf30f894bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:54.192723 25410 tablet_service.cc:1467] Processing CreateTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 (DEFAULT_TABLE table=test-table [id=9189e28cc0d345acb1e9cee68dee594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:54.194101 25410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6ff8ec69d747f59256fcf30f894bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:54.194932 25263 tablet_service.cc:1467] Processing CreateTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 (DEFAULT_TABLE table=test-table [id=9189e28cc0d345acb1e9cee68dee594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:54.196837 25263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6ff8ec69d747f59256fcf30f894bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:54.203779 25650 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Bootstrap starting.
I20260929 16:35:54.206612 25650 tablet_bootstrap.cc:654] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:54.207659 25650 log.cc:824] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:54.209373 25652 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Bootstrap starting.
I20260929 16:35:54.210819 25650 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: No bootstrap required, opened a new log
I20260929 16:35:54.211093 25650 ts_tablet_manager.cc:1397] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 16:35:54.212890 25653 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Bootstrap starting.
I20260929 16:35:54.213781 25652 tablet_bootstrap.cc:654] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:54.214984 25652 log.cc:824] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:54.215773 25653 tablet_bootstrap.cc:654] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:54.216838 25653 log.cc:824] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:54.217969 25652 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: No bootstrap required, opened a new log
I20260929 16:35:54.218225 25652 ts_tablet_manager.cc:1397] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:35:54.219853 25653 tablet_bootstrap.cc:492] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: No bootstrap required, opened a new log
I20260929 16:35:54.220126 25653 ts_tablet_manager.cc:1397] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:35:54.220803 25650 raft_consensus.cc:348] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.221227 25650 raft_consensus.cc:374] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:54.221313 25650 raft_consensus.cc:729] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee28e215f494ab193143e44161047e8, State: Initialized, Role: FOLLOWER
I20260929 16:35:54.221930 25650 consensus_queue.cc:260] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.223585 25617 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
I20260929 16:35:54.227095 25650 ts_tablet_manager.cc:1428] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260929 16:35:54.229012 25653 raft_consensus.cc:348] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.229449 25653 raft_consensus.cc:374] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:54.229554 25653 raft_consensus.cc:729] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a31e0e1b8de41d5a46bd70bf0f3651e, State: Initialized, Role: FOLLOWER
I20260929 16:35:54.230253 25653 consensus_queue.cc:260] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.231943 25653 ts_tablet_manager.cc:1428] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 16:35:54.231734 25652 raft_consensus.cc:348] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.232450 25652 raft_consensus.cc:374] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:54.232556 25652 raft_consensus.cc:729] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e51cda88e4922995232ba64e5172e, State: Initialized, Role: FOLLOWER
I20260929 16:35:54.233160 25652 consensus_queue.cc:260] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.234797 25652 ts_tablet_manager.cc:1428] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260929 16:35:54.235337 25476 heartbeater.cc:503] Master 127.17.205.62:46139 was elected leader, sending a full tablet report...
W20260929 16:35:54.249308 25478 tablet.cc:2369] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:54.319406 25619 tablet.cc:2369] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:35:54.419849 25657 raft_consensus.cc:482] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:54.420176 25657 raft_consensus.cc:504] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.421933 25657 leader_election.cc:288] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901), 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
W20260929 16:35:54.426909 25331 tablet.cc:2369] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:35:54.428066 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.422377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:57845 (local address 127.17.205.3:32901)
0929 16:35:54.422577 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.422583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.423099 (+ 516us) server_negotiation.cc:407] Connection header received
0929 16:35:54.423345 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.423349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.423409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.423483 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:54.423966 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.424680 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.426408 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.427199 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.427378 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.427457 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.427751 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.427878 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:35:54.428504 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.422467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:53365 (local address 127.17.205.2:45241)
0929 16:35:54.422777 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.422784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.423431 (+ 647us) server_negotiation.cc:407] Connection header received
0929 16:35:54.423579 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.423582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.423639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.423749 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:54.424231 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.424887 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.426520 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.427627 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.427773 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.427842 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.428137 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.428255 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 16:35:54.428547 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.422366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45241 (local address 127.17.205.1:53365)
0929 16:35:54.423324 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.423342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.423471 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.423762 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.423767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.423796 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:54.424080 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.424088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.425011 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.425016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.426377 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.426388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.427859 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.427865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.428005 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.428026 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.428103 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.428224 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":658,"threads_started":1}
I20260929 16:35:54.428664 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.422230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:32901 (local address 127.17.205.1:57845)
0929 16:35:54.422982 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:54.423008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:54.423163 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:54.423509 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:54.423513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:54.423537 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:54.423828 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.423837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.424809 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.424815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.426258 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:54.426270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.427411 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.427417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:54.427570 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.427603 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:35:54.428288 (+ 685us) client_negotiation.cc:240] Negotiation successful
0929 16:35:54.428392 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":437,"threads_started":1}
I20260929 16:35:54.429530 25430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6ff8ec69d747f59256fcf30f894bd4" candidate_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e51cda88e4922995232ba64e5172e" is_pre_election: true
I20260929 16:35:54.429584 25571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6ff8ec69d747f59256fcf30f894bd4" candidate_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee28e215f494ab193143e44161047e8" is_pre_election: true
I20260929 16:35:54.429987 25430 raft_consensus.cc:2454] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a31e0e1b8de41d5a46bd70bf0f3651e in term 0.
I20260929 16:35:54.430019 25571 raft_consensus.cc:2454] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a31e0e1b8de41d5a46bd70bf0f3651e in term 0.
I20260929 16:35:54.430594 25219 leader_election.cc:302] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a31e0e1b8de41d5a46bd70bf0f3651e, 5e8e51cda88e4922995232ba64e5172e; no voters:
I20260929 16:35:54.430970 25657 raft_consensus.cc:2781] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:54.431138 25657 raft_consensus.cc:482] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:54.431258 25657 raft_consensus.cc:3037] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:54.433327 25657 raft_consensus.cc:504] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.433837 25657 leader_election.cc:288] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [CANDIDATE]: Term 1 election: Requested vote from peers 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901), 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241)
I20260929 16:35:54.434298 25571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6ff8ec69d747f59256fcf30f894bd4" candidate_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee28e215f494ab193143e44161047e8"
I20260929 16:35:54.434386 25430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6ff8ec69d747f59256fcf30f894bd4" candidate_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e51cda88e4922995232ba64e5172e"
I20260929 16:35:54.434475 25571 raft_consensus.cc:3037] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:54.434577 25430 raft_consensus.cc:3037] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:54.436697 25571 raft_consensus.cc:2454] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a31e0e1b8de41d5a46bd70bf0f3651e in term 1.
I20260929 16:35:54.436748 25430 raft_consensus.cc:2454] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a31e0e1b8de41d5a46bd70bf0f3651e in term 1.
I20260929 16:35:54.437047 25218 leader_election.cc:302] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a31e0e1b8de41d5a46bd70bf0f3651e, 9ee28e215f494ab193143e44161047e8; no voters:
I20260929 16:35:54.437338 25657 raft_consensus.cc:2781] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:54.437769 25657 raft_consensus.cc:686] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 LEADER]: Becoming Leader. State: Replica: 2a31e0e1b8de41d5a46bd70bf0f3651e, State: Running, Role: LEADER
I20260929 16:35:54.438275 25657 consensus_queue.cc:237] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:54.443431 24364 catalog_manager.cc:5495] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1). New cstate: current_term: 1 leader_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } health_report { overall_health: HEALTHY } } }
I20260929 16:35:54.484674 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.479369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.17.205.1:35643)
0929 16:35:54.479565 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.479570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.479591 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:35:54.479774 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.479778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.479858 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.480003 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:54.480395 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.481157 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.482539 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.483000 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.483112 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.483616 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:54.484109 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:54.484241 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.484336 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.484417 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:35:54.492877 25430 raft_consensus.cc:1261] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Refusing update from remote peer 2a31e0e1b8de41d5a46bd70bf0f3651e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:54.492875 25571 raft_consensus.cc:1261] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Refusing update from remote peer 2a31e0e1b8de41d5a46bd70bf0f3651e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:54.493825 25661 consensus_queue.cc:1035] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:54.494246 25657 consensus_queue.cc:1035] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:54.511451 25665 mvcc.cc:205] Tried to move back new op lower bound from 7334706194392899584 to 7334706194183299072. Current Snapshot: MvccSnapshot[applied={T|T < 7334706194392899584}]
I20260929 16:35:54.513964 25664 mvcc.cc:205] Tried to move back new op lower bound from 7334706194392899584 to 7334706194183299072. Current Snapshot: MvccSnapshot[applied={T|T < 7334706194392899584}]
I20260929 16:35:54.514683 25666 mvcc.cc:205] Tried to move back new op lower bound from 7334706194392899584 to 7334706194183299072. Current Snapshot: MvccSnapshot[applied={T|T < 7334706194392899584}]
I20260929 16:35:54.533919 24364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:54.534144 24364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:54.539525 24364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f6e48515165498ea244e5bc2e35e080: Sending DeleteTablet for 3 replicas of tablet de6ff8ec69d747f59256fcf30f894bd4
I20260929 16:35:54.540629 25551 tablet_service.cc:1514] Processing DeleteTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:54 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:44958
I20260929 16:35:54.541103 25410 tablet_service.cc:1514] Processing DeleteTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:54 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:53046
I20260929 16:35:54.541641 25674 tablet_replica.cc:330] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: stopping tablet replica
I20260929 16:35:54.541707 25263 tablet_service.cc:1514] Processing DeleteTablet for tablet de6ff8ec69d747f59256fcf30f894bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:54 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:59510
I20260929 16:35:54.542186 25674 raft_consensus.cc:2229] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:54.542732 25674 raft_consensus.cc:2258] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:54.551425 25674 ts_tablet_manager.cc:1905] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:54.554584 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24332
I20260929 16:35:54.554697 25675 tablet_replica.cc:330] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: stopping tablet replica
I20260929 16:35:54.555151 25675 raft_consensus.cc:2229] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:54.555709 24394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:54.556126 25676 tablet_replica.cc:330] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: stopping tablet replica
I20260929 16:35:54.556596 25676 raft_consensus.cc:2229] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:54.557338 25676 raft_consensus.cc:2258] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:54.557394 25675 raft_consensus.cc:2258] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:54.559630 25676 ts_tablet_manager.cc:1905] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:54.560854 25675 ts_tablet_manager.cc:1905] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:54.566757 25676 ts_tablet_manager.cc:1918] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:54.566984 25676 log.cc:1196] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/de6ff8ec69d747f59256fcf30f894bd4
I20260929 16:35:54.567703 25676 ts_tablet_manager.cc:1939] T de6ff8ec69d747f59256fcf30f894bd4 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting consensus metadata
I20260929 16:35:54.569911 25674 ts_tablet_manager.cc:1918] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:54.570096 25674 log.cc:1196] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/de6ff8ec69d747f59256fcf30f894bd4
I20260929 16:35:54.570626 25674 ts_tablet_manager.cc:1939] T de6ff8ec69d747f59256fcf30f894bd4 P 9ee28e215f494ab193143e44161047e8: Deleting consensus metadata
I20260929 16:35:54.574033 25675 ts_tablet_manager.cc:1918] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:54.574185 25675 log.cc:1196] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/de6ff8ec69d747f59256fcf30f894bd4
I20260929 16:35:54.574842 25675 ts_tablet_manager.cc:1939] T de6ff8ec69d747f59256fcf30f894bd4 P 5e8e51cda88e4922995232ba64e5172e: Deleting consensus metadata
I20260929 16:35:54.580150 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.544050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.17.205.61:38359)
0929 16:35:54.544214 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.544221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.551549 (+ 7328us) server_negotiation.cc:407] Connection header received
0929 16:35:54.552518 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.552525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.552605 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.552670 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.557834 (+ 5164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.559048 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.569274 (+ 10226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.569888 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.569998 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.573655 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.573661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.573666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.573706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.576914 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.577956 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.577962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.577966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.578114 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.578551 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.578557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.578560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.578916 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.579182 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.579560 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.579789 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:35:54.595039 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.553840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.17.205.60:40579)
0929 16:35:54.553998 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.554016 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.580132 (+ 26116us) server_negotiation.cc:407] Connection header received
0929 16:35:54.580358 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.580365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.580446 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.580511 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.582353 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.583592 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.585396 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.586000 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.586133 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.589225 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.589232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.589236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.589273 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.592373 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.593078 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.593083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.593086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.593226 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.593611 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.593614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.593617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.593918 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.594215 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.594464 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.594684 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:35:54.695499 24348 catalog_manager.cc:4841] TS 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643): tablet de6ff8ec69d747f59256fcf30f894bd4 (table test-table [id=9189e28cc0d345acb1e9cee68dee594b]) successfully deleted
I20260929 16:35:54.695653 24351 catalog_manager.cc:4841] TS 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241): tablet de6ff8ec69d747f59256fcf30f894bd4 (table test-table [id=9189e28cc0d345acb1e9cee68dee594b]) successfully deleted
W20260929 16:35:54.695788 24348 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.205.1:35643, user_credentials={real_user=slave}} blocked reactor thread for 126269us
I20260929 16:35:54.696105 24350 catalog_manager.cc:4841] TS 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901): tablet de6ff8ec69d747f59256fcf30f894bd4 (table test-table [id=9189e28cc0d345acb1e9cee68dee594b]) successfully deleted
I20260929 16:35:54.696251 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24332
W20260929 16:35:54.696237 24350 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.205.3:32901, user_credentials={real_user=slave}} blocked reactor thread for 123917us
I20260929 16:35:54.735584 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.717211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.17.205.61:38359)
0929 16:35:54.717578 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.717585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.718419 (+ 834us) server_negotiation.cc:407] Connection header received
0929 16:35:54.718694 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.718701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.718786 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.718861 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.721381 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.722569 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.724640 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.725325 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.725425 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.729289 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.729297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.729302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.729342 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.732767 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.733449 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.733455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.733459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.733597 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.733961 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.733967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.733970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.734346 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.734587 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.734979 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.735197 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 16:35:54.740816 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:54.718128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.17.205.60:40579)
0929 16:35:54.718296 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:54.718303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:54.727422 (+ 9119us) server_negotiation.cc:407] Connection header received
0929 16:35:54.727570 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:54.727576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:54.727660 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:54.727764 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:54.729462 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.730288 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.732088 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:54.732720 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:54.732850 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:54.735723 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:54.735729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:54.735733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:54.735773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:54.738393 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.738969 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.738975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.738979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.739101 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:54.739466 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:54.739470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:54.739472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:54.739756 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:54.739969 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:54.740302 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 16:35:54.740523 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:35:55.522006 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:55.521221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:35309)
0929 16:35:55.521674 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:55.521814 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
W20260929 16:35:55.522341 25617 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:55.527776 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:55.527035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:35881)
0929 16:35:55.527466 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:55.527591 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
W20260929 16:35:55.528259 25476 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:55.530442 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:55.529600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:50985)
0929 16:35:55.530169 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:55.530307 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":396,"threads_started":1}
W20260929 16:35:55.530741 25329 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.040342 25689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f6e48515165498ea244e5bc2e35e080)
I20260929 16:35:56.040768 25690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f6e48515165498ea244e5bc2e35e080)
I20260929 16:35:56.041000 25690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:56.041167 25689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:56.042414 25690 leader_election.cc:288] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359)
I20260929 16:35:56.042511 25689 leader_election.cc:288] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 40511ab026ed43258598a1f16d3ad32b (127.17.205.60:40579)
I20260929 16:35:56.058641 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.042682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54702)
0929 16:35:56.058009 (+ 15327us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.058308 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15131,"threads_started":1}
W20260929 16:35:56.059314 25134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.060173 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.043011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54706)
0929 16:35:56.059705 (+ 16694us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.059918 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16575,"thread_start_us":16411,"threads_started":1}
W20260929 16:35:56.061547 24946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.063898 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.063502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54728)
0929 16:35:56.063669 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.063753 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:35:56.064198 25134 leader_election.cc:334] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.065403 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.064234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54740)
0929 16:35:56.064913 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.065128 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":414,"threads_started":1}
W20260929 16:35:56.065815 24946 leader_election.cc:334] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.068560 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.042883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.17.205.61:38359)
0929 16:35:56.059315 (+ 16432us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:56.059322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:56.059352 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:35:56.059451 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:56.059461 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:56.059572 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:56.059675 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:56.060713 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.062122 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.064798 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.067478 (+ 2680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.067602 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:56.067697 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:56.068096 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 16:35:56.068224 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":16265,"thread_start_us":15997,"threads_started":1}
I20260929 16:35:56.068698 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.042755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:59268)
0929 16:35:56.058374 (+ 15619us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.058402 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:56.058637 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:56.059979 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:56.059984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:56.060016 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:56.060433 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:56.060450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.062316 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.062322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:56.064619 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:56.064631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.067656 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.067663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:56.067917 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:56.067961 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:35:56.068078 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:35:56.068249 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15444,"mutex_wait_us":27,"thread_start_us":16317,"threads_started":1}
I20260929 16:35:56.069479 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.060864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:40579 (local address 127.0.0.1:53894)
0929 16:35:56.061047 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.061065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:56.061187 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:56.061567 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:56.061572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:56.061598 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:56.061925 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:56.061937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.064798 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.064805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:56.067036 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:56.067056 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.068712 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.068720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:56.068931 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:56.068971 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:35:56.069060 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:35:56.069197 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":60}
I20260929 16:35:56.069729 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.060048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.17.205.60:40579)
0929 16:35:56.060932 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:56.060938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:56.061272 (+ 334us) server_negotiation.cc:407] Connection header received
0929 16:35:56.061338 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:56.061343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:56.061415 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:56.062077 (+ 662us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:56.062294 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.064619 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.067230 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:56.068494 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:56.069211 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:56.069280 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:56.069319 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 16:35:56.069409 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":988,"server-negotiator.queue_time_us":747,"thread_start_us":592,"threads_started":1}
I20260929 16:35:56.069741 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40511ab026ed43258598a1f16d3ad32b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78a4809edc9f464daa2f62bee4d3ad47" is_pre_election: true
I20260929 16:35:56.070053 24970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40511ab026ed43258598a1f16d3ad32b in term 1.
I20260929 16:35:56.070377 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78a4809edc9f464daa2f62bee4d3ad47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "40511ab026ed43258598a1f16d3ad32b" is_pre_election: true
I20260929 16:35:56.070536 25133 leader_election.cc:302] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40511ab026ed43258598a1f16d3ad32b, 78a4809edc9f464daa2f62bee4d3ad47; no voters: 6f6e48515165498ea244e5bc2e35e080
I20260929 16:35:56.070750 25162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a4809edc9f464daa2f62bee4d3ad47 in term 1.
I20260929 16:35:56.070909 25690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:35:56.071028 25690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f6e48515165498ea244e5bc2e35e080)
I20260929 16:35:56.071142 25690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:35:56.071296 24946 leader_election.cc:302] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40511ab026ed43258598a1f16d3ad32b, 78a4809edc9f464daa2f62bee4d3ad47; no voters: 6f6e48515165498ea244e5bc2e35e080
I20260929 16:35:56.071647 25689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:35:56.071766 25689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f6e48515165498ea244e5bc2e35e080)
I20260929 16:35:56.071866 25689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:35:56.073361 25690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:56.073966 25690 leader_election.cc:288] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 election: Requested vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359)
I20260929 16:35:56.074156 25689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:56.074569 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.074108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54744)
0929 16:35:56.074325 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.074408 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 16:35:56.074544 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40511ab026ed43258598a1f16d3ad32b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78a4809edc9f464daa2f62bee4d3ad47"
I20260929 16:35:56.074815 24970 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 40511ab026ed43258598a1f16d3ad32b in current term 2: Already voted for candidate 78a4809edc9f464daa2f62bee4d3ad47 in this term.
I20260929 16:35:56.074947 25689 leader_election.cc:288] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 election: Requested vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 40511ab026ed43258598a1f16d3ad32b (127.17.205.60:40579)
I20260929 16:35:56.075459 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.075066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54760)
0929 16:35:56.075236 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.075320 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:35:56.076020 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78a4809edc9f464daa2f62bee4d3ad47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "40511ab026ed43258598a1f16d3ad32b"
I20260929 16:35:56.076171 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.075585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54774)
0929 16:35:56.075745 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.075844 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:35:56.076400 25162 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 78a4809edc9f464daa2f62bee4d3ad47 in current term 2: Already voted for candidate 40511ab026ed43258598a1f16d3ad32b in this term.
W20260929 16:35:56.076342 25134 leader_election.cc:334] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.076589 25134 leader_election.cc:302] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40511ab026ed43258598a1f16d3ad32b; no voters: 6f6e48515165498ea244e5bc2e35e080, 78a4809edc9f464daa2f62bee4d3ad47
I20260929 16:35:56.076871 25690 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 16:35:56.077001 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.076220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:54780)
0929 16:35:56.076716 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.076834 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260929 16:35:56.077143 24946 leader_election.cc:334] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:56.077273 24946 leader_election.cc:302] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78a4809edc9f464daa2f62bee4d3ad47; no voters: 40511ab026ed43258598a1f16d3ad32b, 6f6e48515165498ea244e5bc2e35e080
I20260929 16:35:56.077530 25689 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 16:35:56.523783 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.522980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:33899)
0929 16:35:56.523519 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.523630 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":315,"threads_started":1}
I20260929 16:35:56.529655 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.528908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:44893)
0929 16:35:56.529405 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.529489 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
I20260929 16:35:56.532044 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:56.531382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:51639)
0929 16:35:56.531795 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:56.531885 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":226,"threads_started":1}
I20260929 16:35:57.525247 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.524503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:43847)
0929 16:35:57.524965 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.525065 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":257,"threads_started":1}
I20260929 16:35:57.531097 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.530376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:48747)
0929 16:35:57.530849 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.530929 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260929 16:35:57.533375 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.532749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:52827)
0929 16:35:57.533152 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.533227 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":202,"threads_started":1}
I20260929 16:35:57.701038 25705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:57.701249 25705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:57.701861 25705 leader_election.cc:288] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359)
I20260929 16:35:57.702404 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40511ab026ed43258598a1f16d3ad32b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78a4809edc9f464daa2f62bee4d3ad47" is_pre_election: true
I20260929 16:35:57.702625 24970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40511ab026ed43258598a1f16d3ad32b in term 2.
I20260929 16:35:57.702904 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.702070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55014)
0929 16:35:57.702569 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.702703 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260929 16:35:57.703053 25133 leader_election.cc:302] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40511ab026ed43258598a1f16d3ad32b, 78a4809edc9f464daa2f62bee4d3ad47; no voters:
I20260929 16:35:57.703349 25705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 16:35:57.703505 25705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:57.703608 25705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:35:57.704389 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.704055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55028)
0929 16:35:57.704188 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.704265 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:35:57.704569 25134 leader_election.cc:334] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:57.705848 25705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:57.706327 25705 leader_election.cc:288] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 election: Requested vote from peers 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139), 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359)
I20260929 16:35:57.706756 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.706433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55032)
0929 16:35:57.706563 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.706626 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 16:35:57.706826 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40511ab026ed43258598a1f16d3ad32b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78a4809edc9f464daa2f62bee4d3ad47"
I20260929 16:35:57.707036 24970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:35:57.707701 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.707369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55042)
0929 16:35:57.707514 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.707589 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:35:57.707847 25134 leader_election.cc:334] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
I20260929 16:35:57.709227 24970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40511ab026ed43258598a1f16d3ad32b in term 3.
I20260929 16:35:57.709623 25133 leader_election.cc:302] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40511ab026ed43258598a1f16d3ad32b, 78a4809edc9f464daa2f62bee4d3ad47; no voters: 6f6e48515165498ea244e5bc2e35e080
I20260929 16:35:57.709868 25705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:35:57.710253 25705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [term 3 LEADER]: Becoming Leader. State: Replica: 40511ab026ed43258598a1f16d3ad32b, State: Running, Role: LEADER
I20260929 16:35:57.710645 25705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } }
I20260929 16:35:57.712352 25708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40511ab026ed43258598a1f16d3ad32b. Latest consensus state: current_term: 3 leader_uuid: "40511ab026ed43258598a1f16d3ad32b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:57.712664 25708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:57.713282 25710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:57.716564 25710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:57.717344 25710 catalog_manager.cc:1259] Loaded cluster ID: a879cd01851c48db81d9e05bc6417677
I20260929 16:35:57.717411 25710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:57.718482 25710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:57.719285 25710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Loaded TSK: 0
I20260929 16:35:57.720108 25710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:57.769618 25152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:35:57.776435 25152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:35:57.780491 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.780056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55062)
0929 16:35:57.780251 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.780333 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:35:57.780473 24970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [term 3 FOLLOWER]: Refusing update from remote peer 40511ab026ed43258598a1f16d3ad32b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 16:35:57.781184 25708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 16:35:57.781494 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.781148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55074)
0929 16:35:57.781306 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.781373 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:35:57.781667 25134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b -> Peer 6f6e48515165498ea244e5bc2e35e080 (127.17.205.62:46139): Couldn't send request to peer 6f6e48515165498ea244e5bc2e35e080. Status: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:35:57.784418 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40511ab026ed43258598a1f16d3ad32b. Latest consensus state: current_term: 3 leader_uuid: "40511ab026ed43258598a1f16d3ad32b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:57.784711 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:57.785465 25708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "40511ab026ed43258598a1f16d3ad32b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:57.785730 25708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:57.786339 25713 mvcc.cc:205] Tried to move back new op lower bound from 7334706207862325248 to 7334706207586086912. Current Snapshot: MvccSnapshot[applied={T|T < 7334706207862325248}]
I20260929 16:35:57.787497 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "40511ab026ed43258598a1f16d3ad32b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6f6e48515165498ea244e5bc2e35e080" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46139 } } peers { permanent_uuid: "78a4809edc9f464daa2f62bee4d3ad47" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 38359 } attrs { promote: false } } peers { permanent_uuid: "40511ab026ed43258598a1f16d3ad32b" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 40579 } attrs { promote: false } } }
I20260929 16:35:57.787734 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a4809edc9f464daa2f62bee4d3ad47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:57.802820 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.794208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45241 (local address 127.0.0.1:52886)
0929 16:35:57.795646 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.795663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:57.795778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:57.796294 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:57.796299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:57.796323 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:57.796688 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.796699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.798128 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.798134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.800245 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.800260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.802021 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.802027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.802224 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.802259 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:35:57.802341 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:35:57.802475 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"threads_started":1}
I20260929 16:35:57.803676 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.794364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.17.205.2:45241)
0929 16:35:57.795285 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:57.795291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:57.795884 (+ 593us) server_negotiation.cc:407] Connection header received
0929 16:35:57.795956 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:57.795961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:57.796032 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:57.796110 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:57.796866 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.797962 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.800431 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.801689 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.801816 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.801920 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:57.803183 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0929 16:35:57.803352 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":685,"threads_started":1}
I20260929 16:35:57.805409 25410 tablet_service.cc:1467] Processing CreateTablet for tablet 21be1879733e46508187653e0ed2b222 (DEFAULT_TABLE table=test-table [id=4271b70f95a149548f82c82a264dc43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:57.805820 25410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be1879733e46508187653e0ed2b222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:57.811043 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.794709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:35643 (local address 127.0.0.1:33698)
0929 16:35:57.804272 (+ 9563us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.804288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:57.804365 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:57.805263 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:57.805267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:57.805285 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:57.805386 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.805392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.806764 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.806770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.808561 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.808575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.810176 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.810182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.810421 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.810457 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:35:57.810567 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:35:57.810706 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9447,"mutex_wait_us":20,"thread_start_us":3685,"threads_started":1}
I20260929 16:35:57.811194 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.795069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.17.205.1:35643)
0929 16:35:57.803630 (+ 8561us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:57.803636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:57.804851 (+ 1215us) server_negotiation.cc:407] Connection header received
0929 16:35:57.804927 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:57.804932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:57.805011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:57.805116 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:57.805531 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.806606 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.808731 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.809862 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.809984 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.810088 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:57.810599 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:35:57.810811 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8419,"thread_start_us":8311,"threads_started":1}
I20260929 16:35:57.812197 25263 tablet_service.cc:1467] Processing CreateTablet for tablet 21be1879733e46508187653e0ed2b222 (DEFAULT_TABLE table=test-table [id=4271b70f95a149548f82c82a264dc43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:57.812698 25263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be1879733e46508187653e0ed2b222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:57.813947 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.793875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:32901 (local address 127.0.0.1:43706)
0929 16:35:57.794017 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.794036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:57.794165 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:57.804305 (+ 10140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:57.804310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:57.804337 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:57.804736 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.804750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.809704 (+ 4954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.809710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.811695 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:57.811709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.813188 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.813194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:57.813377 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.813409 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:35:57.813496 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:35:57.813639 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3,"spinlock_wait_cycles":896}
I20260929 16:35:57.813962 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.794094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.17.205.3:32901)
0929 16:35:57.803963 (+ 9869us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:57.803968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:57.803987 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:35:57.804070 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:57.804075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:57.804151 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:57.805526 (+ 1375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:57.805714 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.809539 (+ 3825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.811910 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:57.812986 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:57.813116 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:57.813218 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:57.813542 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:35:57.813675 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9761,"thread_start_us":9671,"threads_started":1}
I20260929 16:35:57.814604 25722 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Bootstrap starting.
I20260929 16:35:57.814713 25551 tablet_service.cc:1467] Processing CreateTablet for tablet 21be1879733e46508187653e0ed2b222 (DEFAULT_TABLE table=test-table [id=4271b70f95a149548f82c82a264dc43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:35:57.815122 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be1879733e46508187653e0ed2b222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:57.816376 25722 tablet_bootstrap.cc:654] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:57.818338 25724 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Bootstrap starting.
I20260929 16:35:57.819519 25722 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: No bootstrap required, opened a new log
I20260929 16:35:57.819695 25722 ts_tablet_manager.cc:1397] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:35:57.819897 25724 tablet_bootstrap.cc:654] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:57.820468 25722 raft_consensus.cc:348] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.821106 25722 raft_consensus.cc:374] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:57.821254 25722 raft_consensus.cc:729] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e51cda88e4922995232ba64e5172e, State: Initialized, Role: FOLLOWER
I20260929 16:35:57.821693 25724 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: No bootstrap required, opened a new log
I20260929 16:35:57.821585 25722 consensus_queue.cc:260] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.821846 25726 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Bootstrap starting.
I20260929 16:35:57.821938 25724 ts_tablet_manager.cc:1397] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:57.822677 25724 raft_consensus.cc:348] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.822947 25724 raft_consensus.cc:374] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:57.823072 25724 raft_consensus.cc:729] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a31e0e1b8de41d5a46bd70bf0f3651e, State: Initialized, Role: FOLLOWER
I20260929 16:35:57.823544 25726 tablet_bootstrap.cc:654] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:57.823379 25724 consensus_queue.cc:260] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.825822 25726 tablet_bootstrap.cc:492] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: No bootstrap required, opened a new log
I20260929 16:35:57.825971 25726 ts_tablet_manager.cc:1397] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:35:57.826743 25726 raft_consensus.cc:348] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.826970 25726 raft_consensus.cc:374] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:57.827061 25726 raft_consensus.cc:729] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee28e215f494ab193143e44161047e8, State: Initialized, Role: FOLLOWER
I20260929 16:35:57.827363 25724 ts_tablet_manager.cc:1428] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:57.827330 25726 consensus_queue.cc:260] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:57.828356 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.827940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:40641)
0929 16:35:57.828109 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.828210 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 16:35:57.828526 25726 ts_tablet_manager.cc:1428] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:57.829185 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.828821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:48493)
0929 16:35:57.828948 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.829039 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:35:57.830219 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.829810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:47619)
0929 16:35:57.829983 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.830064 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:35:57.831547 25328 heartbeater.cc:503] Master 127.17.205.60:40579 was elected leader, sending a full tablet report...
I20260929 16:35:57.831745 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.831249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:37133)
0929 16:35:57.831505 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.831587 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 16:35:57.833796 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.833352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:36211)
0929 16:35:57.833557 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.833649 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:35:57.834904 25722 ts_tablet_manager.cc:1428] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Time spent starting tablet: real 0.015s user 0.002s sys 0.006s
I20260929 16:35:57.835150 25475 heartbeater.cc:503] Master 127.17.205.60:40579 was elected leader, sending a full tablet report...
I20260929 16:35:57.836298 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:57.835854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:36069)
0929 16:35:57.836048 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:57.836139 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 16:35:57.836844 25616 heartbeater.cc:503] Master 127.17.205.60:40579 was elected leader, sending a full tablet report...
I20260929 16:35:58.115417 25730 raft_consensus.cc:482] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:35:58.115685 25730 raft_consensus.cc:504] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:58.117228 25730 leader_election.cc:288] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241), 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:58.123001 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.117622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:52135 (local address 127.17.205.1:35643)
0929 16:35:58.117779 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.117786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.118020 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:35:58.118125 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.118129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.118193 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.118314 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:58.118698 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.119687 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.121413 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.122166 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.122310 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.122385 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.122627 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.122756 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:35:58.123027 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.117523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:35643 (local address 127.17.205.3:52135)
0929 16:35:58.117767 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.117785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:58.117926 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:58.118336 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:58.118341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:58.118364 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:58.118571 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:58.118582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.119817 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.119821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:58.121254 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:58.121267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.122335 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.122340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:58.122489 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.122512 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:35:58.122590 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:35:58.122722 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 16:35:58.123555 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.117672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45781 (local address 127.17.205.2:45241)
0929 16:35:58.117912 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.117920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.118259 (+ 339us) server_negotiation.cc:407] Connection header received
0929 16:35:58.118428 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.118434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.118519 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.118580 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:35:58.119101 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.120095 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.121806 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.122607 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.122770 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.122861 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.123131 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.123288 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:35:58.123631 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.117542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45241 (local address 127.17.205.3:45781)
0929 16:35:58.118159 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.118172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:58.118261 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:58.118735 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:58.118738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:58.118756 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:35:58.118947 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:58.118953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.120241 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.120246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:58.121642 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:58.121656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.122797 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.122803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:58.122973 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.122997 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:35:58.123089 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:35:58.123236 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":5,"thread_start_us":326,"threads_started":1}
I20260929 16:35:58.124150 25283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be1879733e46508187653e0ed2b222" candidate_uuid: "9ee28e215f494ab193143e44161047e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" is_pre_election: true
I20260929 16:35:58.124254 25430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be1879733e46508187653e0ed2b222" candidate_uuid: "9ee28e215f494ab193143e44161047e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e51cda88e4922995232ba64e5172e" is_pre_election: true
I20260929 16:35:58.124483 25430 raft_consensus.cc:2454] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee28e215f494ab193143e44161047e8 in term 0.
I20260929 16:35:58.124558 25283 raft_consensus.cc:2454] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee28e215f494ab193143e44161047e8 in term 0.
I20260929 16:35:58.125008 25507 leader_election.cc:302] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e51cda88e4922995232ba64e5172e, 9ee28e215f494ab193143e44161047e8; no voters:
I20260929 16:35:58.125370 25730 raft_consensus.cc:2781] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:35:58.125501 25730 raft_consensus.cc:482] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:35:58.125597 25730 raft_consensus.cc:3037] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:58.127641 25730 raft_consensus.cc:504] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:58.128207 25730 leader_election.cc:288] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [CANDIDATE]: Term 1 election: Requested vote from peers 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241), 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643)
I20260929 16:35:58.128661 25430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be1879733e46508187653e0ed2b222" candidate_uuid: "9ee28e215f494ab193143e44161047e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e51cda88e4922995232ba64e5172e"
I20260929 16:35:58.128737 25283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be1879733e46508187653e0ed2b222" candidate_uuid: "9ee28e215f494ab193143e44161047e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e"
I20260929 16:35:58.128849 25430 raft_consensus.cc:3037] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:58.128931 25283 raft_consensus.cc:3037] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:58.130755 25430 raft_consensus.cc:2454] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee28e215f494ab193143e44161047e8 in term 1.
I20260929 16:35:58.130949 25283 raft_consensus.cc:2454] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee28e215f494ab193143e44161047e8 in term 1.
I20260929 16:35:58.131147 25507 leader_election.cc:302] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e51cda88e4922995232ba64e5172e, 9ee28e215f494ab193143e44161047e8; no voters:
I20260929 16:35:58.131443 25730 raft_consensus.cc:2781] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:58.131847 25730 raft_consensus.cc:686] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ee28e215f494ab193143e44161047e8, State: Running, Role: LEADER
I20260929 16:35:58.132243 25730 consensus_queue.cc:237] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } }
I20260929 16:35:58.132766 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.132387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:54695)
0929 16:35:58.132555 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.132631 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:35:58.135572 25151 catalog_manager.cc:5495] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ee28e215f494ab193143e44161047e8 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "9ee28e215f494ab193143e44161047e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee28e215f494ab193143e44161047e8" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 } health_report { overall_health: UNKNOWN } } }
I20260929 16:35:58.189239 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.184442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.17.205.3:32901)
0929 16:35:58.184625 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.184630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.184647 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:35:58.184820 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.184824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.184898 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.185017 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:58.185299 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.185977 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.187176 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.187658 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.187756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.188112 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:58.188679 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:58.188791 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.188891 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.188960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:35:58.193199 25430 raft_consensus.cc:1261] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Refusing update from remote peer 9ee28e215f494ab193143e44161047e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:58.193467 25283 raft_consensus.cc:1261] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Refusing update from remote peer 9ee28e215f494ab193143e44161047e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:58.193905 25732 consensus_queue.cc:1035] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e51cda88e4922995232ba64e5172e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:58.194327 25730 consensus_queue.cc:1035] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31e0e1b8de41d5a46bd70bf0f3651e" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:58.196857 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.196276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:46687)
0929 16:35:58.196560 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.196661 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 16:35:58.197870 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.197415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:51145)
0929 16:35:58.197604 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.197708 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:35:58.201541 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.201035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.1:54251)
0929 16:35:58.201266 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.201361 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 16:35:58.204698 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.204030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:35327)
0929 16:35:58.204400 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.204525 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260929 16:35:58.214781 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.214295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.3:58159)
0929 16:35:58.214489 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.214583 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 16:35:58.218031 25151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:58.218179 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.217585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.17.205.2:50905)
0929 16:35:58.217879 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.217973 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 16:35:58.218500 25151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:35:58.226322 25151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b: Sending DeleteTablet for 3 replicas of tablet 21be1879733e46508187653e0ed2b222
I20260929 16:35:58.227186 25551 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be1879733e46508187653e0ed2b222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:58 UTC) from {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:43706
I20260929 16:35:58.227202 25410 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be1879733e46508187653e0ed2b222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:58 UTC) from {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:52886
I20260929 16:35:58.227892 25409 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be1879733e46508187653e0ed2b222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:58 UTC) from {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:52886
I20260929 16:35:58.228875 25263 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be1879733e46508187653e0ed2b222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:35:58 UTC) from {username='oryx', principal='oryx/127.17.205.60@KRBTEST.COM'} at 127.0.0.1:33698
I20260929 16:35:58.234359 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 25112
I20260929 16:35:58.242367 25748 tablet_replica.cc:330] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: stopping tablet replica
I20260929 16:35:58.242573 25748 raft_consensus.cc:2229] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:58.242899 25748 raft_consensus.cc:2258] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:58.245944 25748 ts_tablet_manager.cc:1905] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:58.246781 25746 tablet_replica.cc:330] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: stopping tablet replica
I20260929 16:35:58.247093 25746 raft_consensus.cc:2229] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:35:58.247794 25746 raft_consensus.cc:2258] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:58.248971 25746 ts_tablet_manager.cc:1905] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:58.254570 25747 tablet_replica.cc:330] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: stopping tablet replica
I20260929 16:35:58.254801 25747 raft_consensus.cc:2229] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:35:58.255134 25747 raft_consensus.cc:2258] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:35:58.258594 25748 ts_tablet_manager.cc:1918] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:58.258734 25748 log.cc:1196] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/21be1879733e46508187653e0ed2b222
I20260929 16:35:58.259162 25748 ts_tablet_manager.cc:1939] T 21be1879733e46508187653e0ed2b222 P 2a31e0e1b8de41d5a46bd70bf0f3651e: Deleting consensus metadata
I20260929 16:35:58.260366 25133 catalog_manager.cc:4841] TS 2a31e0e1b8de41d5a46bd70bf0f3651e (127.17.205.1:35643): tablet 21be1879733e46508187653e0ed2b222 (table test-table [id=4271b70f95a149548f82c82a264dc43d]) successfully deleted
I20260929 16:35:58.260636 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.236570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.17.205.1:35643)
0929 16:35:58.236751 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.236757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.237844 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 16:35:58.237915 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.237921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.238006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.238092 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:58.241125 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.242389 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.244900 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.245507 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.245624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.252767 (+ 7143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:58.252786 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:58.252806 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:58.252849 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:58.256677 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.257606 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.257617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.257621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.257758 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.258182 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.258187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.258189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.258646 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:58.258919 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.259547 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.259773 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:35:58.261977 25746 ts_tablet_manager.cc:1918] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:58.262061 25323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:58.262105 25746 log.cc:1196] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/21be1879733e46508187653e0ed2b222
I20260929 16:35:58.262586 25746 ts_tablet_manager.cc:1939] T 21be1879733e46508187653e0ed2b222 P 9ee28e215f494ab193143e44161047e8: Deleting consensus metadata
I20260929 16:35:58.264171 25747 ts_tablet_manager.cc:1905] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:35:58.267827 25135 catalog_manager.cc:4841] TS 9ee28e215f494ab193143e44161047e8 (127.17.205.3:32901): tablet 21be1879733e46508187653e0ed2b222 (table test-table [id=4271b70f95a149548f82c82a264dc43d]) successfully deleted
I20260929 16:35:58.271667 25747 ts_tablet_manager.cc:1918] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:35:58.271803 25747 log.cc:1196] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/21be1879733e46508187653e0ed2b222
I20260929 16:35:58.272207 25747 ts_tablet_manager.cc:1939] T 21be1879733e46508187653e0ed2b222 P 5e8e51cda88e4922995232ba64e5172e: Deleting consensus metadata
I20260929 16:35:58.273384 25136 catalog_manager.cc:4841] TS 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241): tablet 21be1879733e46508187653e0ed2b222 (table test-table [id=4271b70f95a149548f82c82a264dc43d]) successfully deleted
W20260929 16:35:58.273867 25136 catalog_manager.cc:4808] TS 5e8e51cda88e4922995232ba64e5172e (127.17.205.2:45241): delete failed for tablet 21be1879733e46508187653e0ed2b222 because the tablet was not found. No further retry: Not found: Tablet not found: 21be1879733e46508187653e0ed2b222
I20260929 16:35:58.379287 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.378486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55082)
0929 16:35:58.378964 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.379065 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260929 16:35:58.380736 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.380299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55094)
0929 16:35:58.380464 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.380558 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:35:58.384231 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 25112
I20260929 16:35:58.402776 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 25339
I20260929 16:35:58.415421 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.403622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.17.205.2:45241)
0929 16:35:58.403825 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.403831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.403849 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:35:58.403977 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.403982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.404048 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.404136 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:58.405342 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.406036 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.407385 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.407813 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.407922 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.410251 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:58.410266 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:58.410281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:58.410312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:58.412902 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.413511 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.413519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.413521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.413650 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.413999 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.414002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.414004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.414339 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:58.414604 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.414885 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.415111 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:35:58.416061 25470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:58.544564 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 25339
I20260929 16:35:58.565201 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 25488
I20260929 16:35:58.580991 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.565956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.17.205.3:32901)
0929 16:35:58.566163 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:58.566169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:58.566192 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:35:58.566316 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:58.566322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:58.566429 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:58.566562 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:58.567974 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.569176 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.570572 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:58.571252 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:58.571401 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:58.573834 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:58.573853 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:58.573875 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:58.573928 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:58.578011 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.578739 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.578753 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.578757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.578947 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:58.579343 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:58.579349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:58.579352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:58.579849 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:58.580182 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:58.580343 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:35:58.580587 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:35:58.581837 25611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:58.706689 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 25488
I20260929 16:35:58.723888 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24825
I20260929 16:35:58.724965 24991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:58.815430 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.814881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55096)
0929 16:35:58.815074 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.815203 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:35:58.816481 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.816123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46139 (local address 127.0.0.1:55112)
0929 16:35:58.816266 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.816349 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 16:35:58.838964 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24825
I20260929 16:35:58.857921 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 24925
I20260929 16:35:58.858577 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:58.858132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:38359 (local address 127.0.0.1:59278)
0929 16:35:58.858300 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:58.858397 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:35:58.858776 25133 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40511ab026ed43258598a1f16d3ad32b -> Peer 78a4809edc9f464daa2f62bee4d3ad47 (127.17.205.61:38359): Couldn't send request to peer 78a4809edc9f464daa2f62bee4d3ad47. Status: Network error: Client connection negotiation failed: client connection to 127.17.205.61:38359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:35:58.858932 25182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:35:58.976325 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 24925
2026-09-29T16:35:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11156 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:59 dist-test-slave-3h01 krb5kdc[25759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:59 dist-test-slave-3h01 krb5kdc[25759](info): set up 2 sockets
Sep 29 16:35:59 dist-test-slave-3h01 krb5kdc[25759](info): commencing operation
krb5kdc: starting...
W20260929 16:35:59.983769 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:36:00 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab.
Entry for principal oryx/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab.
I20260929 16:36:00.191533 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36151
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab
--principal=oryx/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:00.511847 25772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:00.512204 25772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:00.512338 25772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:00.519660 25772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:00.519795 25772 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:36:00.519857 25772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:00.519948 25772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:00.520021 25772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:00.520105 25772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:00.530046 25772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:36151
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.62.keytab
--principal=oryx/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:00.531394 25772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:00.532734 25772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:00.544288 25777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:00.544250 25778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:00.544209 25780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:00.545123 25772 server_base.cc:1034] running on GCE node
I20260929 16:36:00.545840 25772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:00.547238 25772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:00.548452 25772 hybrid_clock.cc:648] HybridClock initialized: now 1790699760548379 us; error 55 us; skew 500 ppm
I20260929 16:36:00.548756 25772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:00 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699760, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:00.551717 25772 init.cc:377] Logged in from keytab as oryx/127.17.205.62@KRBTEST.COM (short username oryx)
I20260929 16:36:00.553054 25772 webserver.cc:460] Webserver started at http://127.17.205.62:45915/ using document root <none> and password file <none>
I20260929 16:36:00.553740 25772 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:00.553839 25772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:00.554140 25772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:00.557403 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "22f6f65826ee483285ee73db2c55eebd"
format_stamp: "Formatted at 2026-09-29 16:36:00 on dist-test-slave-3h01"
server_key: "110d38dec30a65f5963b59ed4340ef67"
server_key_iv: "14290d99ee85e0515501073771ae89f6"
server_key_version: "encryptionkey@0"
I20260929 16:36:00.558195 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "22f6f65826ee483285ee73db2c55eebd"
format_stamp: "Formatted at 2026-09-29 16:36:00 on dist-test-slave-3h01"
server_key: "110d38dec30a65f5963b59ed4340ef67"
server_key_iv: "14290d99ee85e0515501073771ae89f6"
server_key_version: "encryptionkey@0"
I20260929 16:36:00.563505 25772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:36:00.567042 25787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:00.568418 25772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:36:00.568571 25772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "22f6f65826ee483285ee73db2c55eebd"
format_stamp: "Formatted at 2026-09-29 16:36:00 on dist-test-slave-3h01"
server_key: "110d38dec30a65f5963b59ed4340ef67"
server_key_iv: "14290d99ee85e0515501073771ae89f6"
server_key_version: "encryptionkey@0"
I20260929 16:36:00.568763 25772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:00.594836 25772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:00.598892 25772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:00.599136 25772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:00.617941 25772 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:36151
I20260929 16:36:00.617960 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:36151 every 8 connection(s)
I20260929 16:36:00.619280 25772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:00.621615 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 25772
I20260929 16:36:00.621788 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:36:00.622397 18228 external_mini_cluster.cc:1428] Setting key 3b2712f4e9204fdfbc1173c7696ac54d
I20260929 16:36:00.623122 25849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:36:00 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:00.636788 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Bootstrap starting.
I20260929 16:36:00.640774 25849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:00.641937 25849 log.cc:824] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:00.645216 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: No bootstrap required, opened a new log
I20260929 16:36:00.645246 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:00.624230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.17.205.62:36151)
0929 16:36:00.624961 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:00.624988 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:00.625037 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:36:00.626687 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:00.626714 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:00.627268 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:00.627717 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:00.628910 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:00.630900 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:00.632178 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:00.633152 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:00.633336 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:00.637265 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:00.637275 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:00.637295 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:00.637344 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:00.640917 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:00.641565 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:00.641573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:00.641582 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:00.641790 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:00.642108 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:00.642113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:00.642116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:00.642754 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:00.643021 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:00.643494 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:36:00.643909 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":319,"threads_started":1}
I20260929 16:36:00.652463 25849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } }
I20260929 16:36:00.652812 25849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:00.652881 25849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f6f65826ee483285ee73db2c55eebd, State: Initialized, Role: FOLLOWER
I20260929 16:36:00.653357 25849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } }
I20260929 16:36:00.653561 25849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:00.653745 25849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:00.653963 25849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:00.656137 25849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } }
I20260929 16:36:00.656613 25849 leader_election.cc:302] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f6f65826ee483285ee73db2c55eebd; no voters:
I20260929 16:36:00.657135 25849 leader_election.cc:288] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:36:00.657251 25854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:00.658138 25854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [term 1 LEADER]: Becoming Leader. State: Replica: 22f6f65826ee483285ee73db2c55eebd, State: Running, Role: LEADER
I20260929 16:36:00.658618 25854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } }
I20260929 16:36:00.659058 25849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:00.661191 25855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f6f65826ee483285ee73db2c55eebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } } }
I20260929 16:36:00.661566 25855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:00.661266 25856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f6f65826ee483285ee73db2c55eebd. Latest consensus state: current_term: 1 leader_uuid: "22f6f65826ee483285ee73db2c55eebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f6f65826ee483285ee73db2c55eebd" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36151 } } }
I20260929 16:36:00.661809 25856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:00.663640 25862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:00.668821 25862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:00.677084 25862 catalog_manager.cc:1347] Generated new cluster ID: dae57875a39b4673a882e6d1e53981c3
I20260929 16:36:00.677206 25862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:00.687376 25862 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:36:00.688760 25862 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:00.697057 25862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Generated new TSK 0
I20260929 16:36:00.697926 25862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
I20260929 16:36:00.746426 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:36151
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:01.067783 25875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:01.068070 25875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:01.068202 25875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:01.075673 25875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:01.075788 25875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:01.075907 25875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:36:01.076010 25875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:01.085935 25875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:01.087342 25875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:01.088690 25875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:01.101333 25881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:01.101408 25880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:01.101369 25883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:01.104645 25875 server_base.cc:1034] running on GCE node
I20260929 16:36:01.105247 25875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:01.106212 25875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:01.107410 25875 hybrid_clock.cc:648] HybridClock initialized: now 1790699761107331 us; error 65 us; skew 500 ppm
I20260929 16:36:01.107689 25875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:01 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699761, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:01.110661 25875 init.cc:377] Logged in from keytab as oryx/127.17.205.1@KRBTEST.COM (short username oryx)
I20260929 16:36:01.112170 25875 webserver.cc:460] Webserver started at http://127.17.205.1:35827/ using document root <none> and password file <none>
I20260929 16:36:01.112855 25875 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:01.112939 25875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:01.113206 25875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:01.116199 25875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "c8406dfa745ce14042da9efda12e750c"
server_key_iv: "36eae072782ecfa4201e0b38cddd501a"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.116948 25875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "c8406dfa745ce14042da9efda12e750c"
server_key_iv: "36eae072782ecfa4201e0b38cddd501a"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.122059 25875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:36:01.125627 25890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:01.126982 25875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:36:01.127136 25875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "c8406dfa745ce14042da9efda12e750c"
server_key_iv: "36eae072782ecfa4201e0b38cddd501a"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.127384 25875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:01.148080 25875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:01.151829 25875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:01.152070 25875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:01.154610 25875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:01.154717 25875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:01.154863 25875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:01.154924 25875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:36:01 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699761, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:01.180184 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.155677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43653 (local address 127.17.205.62:36151)
0929 16:36:01.157888 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:01.157894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:01.157917 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:36:01.159831 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:01.159838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:01.159932 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:01.160027 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:01.162722 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.163894 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.166734 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.167351 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.167497 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.172831 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:01.172840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:01.172846 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:01.172893 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:01.176378 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.177424 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.177429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.177433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.177582 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.178157 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.178162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.178165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.178566 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:01.178701 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:01.179532 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 16:36:01.179756 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094,"thread_start_us":1990,"threads_started":1}
I20260929 16:36:01.181293 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.156123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.1:43653)
0929 16:36:01.157181 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:01.157239 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:01.159415 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:01.160699 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:01.160712 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:01.161513 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:01.162495 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.162522 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.164106 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.164112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.166567 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.166586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.167647 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.167656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.167936 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.168852 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:01.168893 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:01.172549 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:01.176530 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.176540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.176563 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.177237 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.177709 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.177719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.177723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.178034 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.178715 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:01.178725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:01.179112 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 16:36:01.179461 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 16:36:01.180025 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":379,"threads_started":1}
I20260929 16:36:01.200533 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.187448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.1:39875)
0929 16:36:01.187961 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:01.187976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:01.188119 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:01.188627 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:01.188631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:01.189054 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:01.189324 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.189333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.190266 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.190272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.191292 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.191304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.191999 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.192006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.192160 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.192759 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:01.192783 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:01.193786 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:01.197563 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.197568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.197571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.197966 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.198374 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.198377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.198379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.198516 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.199134 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:01.199139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:01.199482 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 16:36:01.199687 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:36:01.200051 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":278,"threads_started":1}
I20260929 16:36:01.201251 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.187381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:39875 (local address 127.17.205.62:36151)
0929 16:36:01.187556 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:01.187562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:01.188065 (+ 503us) server_negotiation.cc:407] Connection header received
0929 16:36:01.188232 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:01.188237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:01.188316 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:01.188405 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:01.189465 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.190149 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.191408 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.191826 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.191928 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.194019 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:01.194025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:01.194030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:01.194059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:01.197388 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.198088 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.198094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.198099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.198279 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.198599 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.198604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.198608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.198964 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:01.199144 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:01.200848 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0929 16:36:01.201029 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:36:01.204093 25875 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42977
I20260929 16:36:01.204119 26010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42977 every 8 connection(s)
I20260929 16:36:01.205351 25875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:01.207122 26011 heartbeater.cc:347] Connected to a master server at 127.17.205.62:36151
I20260929 16:36:01.207383 26011 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:01.207507 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 25875
I20260929 16:36:01.207712 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:36:01.208174 26011 heartbeater.cc:511] Master 127.17.205.62:36151 requested a full tablet report, sending...
I20260929 16:36:01.208508 18228 external_mini_cluster.cc:1428] Setting key e26a47d05e76cb6a68f0b4d78b045f26
I20260929 16:36:01.210239 25804 ts_manager.cc:194] Registered new tserver with Master: 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977)
I20260929 16:36:01.211697 25804 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:43653
WARNING: no policy specified for oryx/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
I20260929 16:36:01.253834 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:36151
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:01.571554 26015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:01.571820 26015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:01.571946 26015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:01.579391 26015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:01.579506 26015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:01.579630 26015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:36:01.579732 26015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:01.589463 26015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:01.590751 26015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:01.592109 26015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:01.604908 26021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:01.605347 26023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:01.605372 26020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:01.605913 26015 server_base.cc:1034] running on GCE node
I20260929 16:36:01.606526 26015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:01.607489 26015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:01.608736 26015 hybrid_clock.cc:648] HybridClock initialized: now 1790699761608654 us; error 55 us; skew 500 ppm
I20260929 16:36:01.609040 26015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:01 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699761, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:01.615190 26015 init.cc:377] Logged in from keytab as oryx/127.17.205.2@KRBTEST.COM (short username oryx)
I20260929 16:36:01.616830 26015 webserver.cc:460] Webserver started at http://127.17.205.2:33125/ using document root <none> and password file <none>
I20260929 16:36:01.617690 26015 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:01.617807 26015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:01.618175 26015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:01.622187 26015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "150bc1e7d32847dca146b8c5c994da44"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "053bd26cefdcc299151253f3c867ffb0"
server_key_iv: "522d7eac22841684c580cd43f8b27bcd"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.623194 26015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "150bc1e7d32847dca146b8c5c994da44"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "053bd26cefdcc299151253f3c867ffb0"
server_key_iv: "522d7eac22841684c580cd43f8b27bcd"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.629878 26015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:36:01.634332 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:01.636044 26015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:36:01.636220 26015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "150bc1e7d32847dca146b8c5c994da44"
format_stamp: "Formatted at 2026-09-29 16:36:01 on dist-test-slave-3h01"
server_key: "053bd26cefdcc299151253f3c867ffb0"
server_key_iv: "522d7eac22841684c580cd43f8b27bcd"
server_key_version: "encryptionkey@0"
I20260929 16:36:01.636463 26015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:01.647559 26015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:01.651381 26015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:01.651623 26015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:01.654170 26015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:01.654273 26015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:01.654408 26015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:01.654479 26015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:36:01 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699761, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:01.682919 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.655475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.2:54725)
0929 16:36:01.656251 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:01.656290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:01.658078 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:01.658921 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:01.658934 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:01.659917 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:01.660965 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.661006 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.662330 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.662339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.664812 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.664829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.666441 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.666448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.666789 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.669068 (+ 2279us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:01.669145 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:01.673252 (+ 4107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:01.677107 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.677118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.677138 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.677786 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.678292 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.678298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.678302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.678590 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.679317 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:01.679327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:01.679994 (+ 667us) client_negotiation.cc:769] Sending connection context
0929 16:36:01.680344 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 16:36:01.681038 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260929 16:36:01.684530 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.654717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:54725 (local address 127.17.205.62:36151)
0929 16:36:01.654878 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:01.654882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:01.656402 (+ 1520us) server_negotiation.cc:407] Connection header received
0929 16:36:01.658381 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:01.658387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:01.658476 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:01.658583 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:01.661206 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.662145 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.665036 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.665732 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.666081 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.673719 (+ 7638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:01.673727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:01.673731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:01.673769 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:01.676951 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.677943 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.677949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.677952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.678137 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.678746 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.678751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.678755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.679126 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:01.679315 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:01.683882 (+ 4567us) server_negotiation.cc:299] Negotiation successful
0929 16:36:01.684161 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:36:01.713222 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.696876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:41315 (local address 127.17.205.62:36151)
0929 16:36:01.697089 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:01.697095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:01.697560 (+ 465us) server_negotiation.cc:407] Connection header received
0929 16:36:01.697771 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:01.697778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:01.697861 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:01.697973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:01.699656 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.700792 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.703241 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.703834 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.703954 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.706680 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:01.706686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:01.706691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:01.706735 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:01.710053 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.710923 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.710929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.710932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.711094 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:01.711527 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:01.711532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:01.711535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:01.711990 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:01.712149 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:01.712650 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 16:36:01.712887 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:36:01.713382 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:01.696891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.2:41315)
0929 16:36:01.697451 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:01.697471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:01.697634 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:01.698168 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:01.698173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:01.698929 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:01.699441 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.699459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.701038 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.701043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.703010 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:01.703024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:01.704301 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:01.704308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:01.704499 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:01.705281 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:01.705307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:01.706426 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:01.710216 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.710222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.710227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.710800 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.711199 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:01.711205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:01.711245 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:01.711409 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:01.712127 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:01.712133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:01.712350 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 16:36:01.712523 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:36:01.712878 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":826,"threads_started":1}
I20260929 16:36:01.724779 26015 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38853
I20260929 16:36:01.724800 26149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38853 every 8 connection(s)
I20260929 16:36:01.725996 26015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:01.727958 26150 heartbeater.cc:347] Connected to a master server at 127.17.205.62:36151
I20260929 16:36:01.728200 26150 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:01.728843 26150 heartbeater.cc:511] Master 127.17.205.62:36151 requested a full tablet report, sending...
I20260929 16:36:01.730180 25804 ts_manager.cc:194] Registered new tserver with Master: 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853)
I20260929 16:36:01.730969 25804 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:54725
I20260929 16:36:01.735499 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26015
I20260929 16:36:01.735639 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:36:01.736002 18228 external_mini_cluster.cc:1428] Setting key 2f11f846c5f6e8b33f3879d9e24dd59a
WARNING: no policy specified for oryx/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
I20260929 16:36:01.778400 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:36151
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:02.094954 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:02.095274 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:02.095413 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:02.102746 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:02.102873 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:02.102999 26154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:36:02.103116 26154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:02.113250 26154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.205.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:02.114514 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:02.115916 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:02.128073 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:02.128212 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:02.128212 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:02.128969 26154 server_base.cc:1034] running on GCE node
I20260929 16:36:02.129531 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:02.130385 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:02.131593 26154 hybrid_clock.cc:648] HybridClock initialized: now 1790699762131515 us; error 56 us; skew 500 ppm
I20260929 16:36:02.131894 26154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:02 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699762, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:02.134975 26154 init.cc:377] Logged in from keytab as oryx/127.17.205.3@KRBTEST.COM (short username oryx)
I20260929 16:36:02.136374 26154 webserver.cc:460] Webserver started at http://127.17.205.3:43181/ using document root <none> and password file <none>
I20260929 16:36:02.137055 26154 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:02.137149 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:02.137445 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:02.140516 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
format_stamp: "Formatted at 2026-09-29 16:36:02 on dist-test-slave-3h01"
server_key: "87e4327fb509e30bee222fe1f24d5d3e"
server_key_iv: "92d1af032fc1c1e695dc1608a1988eaa"
server_key_version: "encryptionkey@0"
I20260929 16:36:02.141330 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
format_stamp: "Formatted at 2026-09-29 16:36:02 on dist-test-slave-3h01"
server_key: "87e4327fb509e30bee222fe1f24d5d3e"
server_key_iv: "92d1af032fc1c1e695dc1608a1988eaa"
server_key_version: "encryptionkey@0"
I20260929 16:36:02.146451 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:36:02.149876 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:02.151161 26154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:02.151338 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
format_stamp: "Formatted at 2026-09-29 16:36:02 on dist-test-slave-3h01"
server_key: "87e4327fb509e30bee222fe1f24d5d3e"
server_key_iv: "92d1af032fc1c1e695dc1608a1988eaa"
server_key_version: "encryptionkey@0"
I20260929 16:36:02.151533 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:02.193186 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:02.197119 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:02.197373 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:02.200043 26154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:02.200131 26154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:02.200268 26154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:02.200376 26154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 16:36:02.214847 26011 heartbeater.cc:503] Master 127.17.205.62:36151 was elected leader, sending a full tablet report...
Sep 29 16:36:02 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699762, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:02.223709 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:02.200556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:37545 (local address 127.17.205.62:36151)
0929 16:36:02.200775 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:02.200781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:02.202367 (+ 1586us) server_negotiation.cc:407] Connection header received
0929 16:36:02.205263 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:02.205270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:02.205360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:02.205443 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:02.208259 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.208976 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.211301 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.211809 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.211886 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:02.217252 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:02.217257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:02.217260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:02.217291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:02.219969 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:02.220940 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:02.220946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:02.220949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:02.221132 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:02.221576 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:02.221581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:02.221584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:02.221847 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:02.221972 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:02.223044 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0929 16:36:02.223323 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:36:02.224776 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:02.201248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.3:37545)
0929 16:36:02.202177 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:02.202237 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:02.204868 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:02.206041 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:02.206054 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:02.206909 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:02.208049 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:02.208080 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.209147 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.209155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:02.211094 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:02.211112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.212256 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.212264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:02.212554 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:02.213359 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:02.213395 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:02.217045 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:02.220102 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:02.220113 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:02.220134 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:02.220756 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:02.221243 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:02.221246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:02.221249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:02.221456 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:02.221968 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:02.221982 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:02.222496 (+ 514us) client_negotiation.cc:769] Sending connection context
0929 16:36:02.222856 (+ 360us) client_negotiation.cc:240] Negotiation successful
0929 16:36:02.223355 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":386,"threads_started":1}
I20260929 16:36:02.243611 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:02.229932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:40997 (local address 127.17.205.62:36151)
0929 16:36:02.230117 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:02.230124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:02.230711 (+ 587us) server_negotiation.cc:407] Connection header received
0929 16:36:02.230922 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:02.230928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:02.231021 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:02.231086 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:02.232298 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.233470 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.234756 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.235436 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.235560 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:02.237816 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:02.237824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:02.237829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:02.237869 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:02.241016 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:02.241840 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:02.241844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:02.241847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:02.241963 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:02.242291 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:02.242294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:02.242296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:02.242532 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:02.242634 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:02.243050 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:36:02.243259 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:36:02.243885 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:02.229977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.17.205.3:40997)
0929 16:36:02.230618 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:02.230636 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:02.230776 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:02.231285 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:02.231290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:02.231835 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:02.232160 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:02.232171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.233625 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.233629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:02.234641 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:02.234651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.235625 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.235632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:02.235822 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:02.236429 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:02.236452 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:02.237580 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:02.241166 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:02.241172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:02.241177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:02.241735 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:02.242053 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:02.242058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:02.242062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:02.242209 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:02.242621 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:02.242625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:02.242824 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 16:36:02.242985 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:36:02.243376 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":349,"threads_started":1}
I20260929 16:36:02.252007 26154 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:37551
I20260929 16:36:02.252027 26288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:37551 every 8 connection(s)
I20260929 16:36:02.253165 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:02.255038 26289 heartbeater.cc:347] Connected to a master server at 127.17.205.62:36151
I20260929 16:36:02.255264 26289 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:02.255946 26289 heartbeater.cc:511] Master 127.17.205.62:36151 requested a full tablet report, sending...
I20260929 16:36:02.257164 25804 ts_manager.cc:194] Registered new tserver with Master: 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:02.257925 25804 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:37545
I20260929 16:36:02.258363 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26154
I20260929 16:36:02.258503 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:36:02.258913 18228 external_mini_cluster.cc:1428] Setting key adce18559f23c921c40805cbd8677714
I20260929 16:36:02.261648 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:36:02 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699762, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:36:02 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:02 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:02.733490 26150 heartbeater.cc:503] Master 127.17.205.62:36151 was elected leader, sending a full tablet report...
I20260929 16:36:03.260499 26289 heartbeater.cc:503] Master 127.17.205.62:36151 was elected leader, sending a full tablet report...
W20260929 16:36:03.266654 26297 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.964s user 0.008s sys 0.004s
W20260929 16:36:03.267702 25899 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:02.297670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.17.205.62:36151)
0929 16:36:02.297877 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:02.297883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:02.298154 (+ 271us) server_negotiation.cc:407] Connection header received
0929 16:36:02.298301 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:02.298305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:02.298385 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:02.298496 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:02.299725 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.300452 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.301632 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:02.302048 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:02.302149 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.267434 (+965285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37920: BlockingRecv error: recv got EOF from 127.0.0.1:37920 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:36:03 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699762, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:03.289677 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.271274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.17.205.62:36151)
0929 16:36:03.271467 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.271471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.271851 (+ 380us) server_negotiation.cc:407] Connection header received
0929 16:36:03.272050 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.272055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.272138 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.272242 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:03.273592 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.274297 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.275589 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.276026 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.276119 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.284701 (+ 8582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:03.284706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:03.284709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:03.284750 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:03.287301 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:03.287998 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:03.288032 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:03.288034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:03.288167 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:03.288537 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:03.288539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:03.288541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:03.288804 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:03.289047 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.289276 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.289442 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:36:03.293541 25804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:36:03.300707 25804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:03.334148 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.313179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37551 (local address 127.0.0.1:43846)
0929 16:36:03.325647 (+ 12468us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.325666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.325789 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.326700 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.326704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.326728 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.327141 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.327154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.328927 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.328932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.331174 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.331185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.333244 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.333250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.333416 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.333465 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.333572 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.333755 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12344,"mutex_wait_us":1,"thread_start_us":10607,"threads_started":1}
I20260929 16:36:03.334846 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.313539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.17.205.3:37551)
0929 16:36:03.324889 (+ 11350us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.324900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.325932 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 16:36:03.326031 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.326037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.326274 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.326453 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.327347 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.328765 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.331339 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.332772 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.332896 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.333112 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.334402 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.334573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11168,"thread_start_us":11637,"threads_started":1}
I20260929 16:36:03.336656 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.312672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42977 (local address 127.0.0.1:56216)
0929 16:36:03.323291 (+ 10619us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.323355 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.323554 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.326047 (+ 2493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.326054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.326100 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.326535 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.326548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.330057 (+ 3509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.330062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.333298 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.333310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.335379 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.335598 (+ 219us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.335799 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.335998 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.336105 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.336246 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10265,"mutex_wait_us":161,"thread_start_us":12745,"threads_started":1}
I20260929 16:36:03.337180 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.312830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.17.205.1:42977)
0929 16:36:03.325345 (+ 12515us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.325354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.325387 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:36:03.325492 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.325499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.325832 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.326039 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.326740 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.329921 (+ 3181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.333467 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.335136 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.335327 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.335533 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.336827 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.336969 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12356,"thread_start_us":12216,"threads_started":1}
I20260929 16:36:03.339051 26219 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 (DEFAULT_TABLE table=test-table [id=d16ff597da504ca09019a75610bcb197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.339758 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.326404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.0.0.1:59660)
0929 16:36:03.331543 (+ 5139us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.331561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.331729 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.332375 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.332379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.332404 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.332965 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.332978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.335442 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.335447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.337556 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.337571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.338999 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.339005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.339163 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.339196 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.339305 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.339429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5021,"thread_start_us":5398,"threads_started":1}
I20260929 16:36:03.339830 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.326064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.17.205.2:38853)
0929 16:36:03.329860 (+ 3796us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.329868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.331669 (+ 1801us) server_negotiation.cc:407] Connection header received
0929 16:36:03.331901 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.331910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.332160 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.332353 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.333249 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.335288 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.337747 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.338805 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.338931 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.339142 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.339346 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.339509 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3650,"thread_start_us":3513,"threads_started":1}
I20260929 16:36:03.340873 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdfa7edc9994dbc9af734eee30949e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.340909 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 (DEFAULT_TABLE table=test-table [id=d16ff597da504ca09019a75610bcb197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.342252 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdfa7edc9994dbc9af734eee30949e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.342725 26080 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 (DEFAULT_TABLE table=test-table [id=d16ff597da504ca09019a75610bcb197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.343873 26080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdfa7edc9994dbc9af734eee30949e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.355010 26314 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Bootstrap starting.
I20260929 16:36:03.358811 26314 tablet_bootstrap.cc:654] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.359688 26316 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Bootstrap starting.
I20260929 16:36:03.360049 26314 log.cc:824] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:03.360190 26315 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Bootstrap starting.
I20260929 16:36:03.363814 26315 tablet_bootstrap.cc:654] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.363813 26316 tablet_bootstrap.cc:654] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.365042 26315 log.cc:824] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:03.365042 26316 log.cc:824] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:03.365566 26314 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: No bootstrap required, opened a new log
I20260929 16:36:03.365886 26314 ts_tablet_manager.cc:1397] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260929 16:36:03.373984 26315 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: No bootstrap required, opened a new log
I20260929 16:36:03.374296 26315 ts_tablet_manager.cc:1397] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260929 16:36:03.374176 26314 raft_consensus.cc:348] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.374580 26314 raft_consensus.cc:374] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.374684 26314 raft_consensus.cc:729] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.374763 26316 tablet_bootstrap.cc:492] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: No bootstrap required, opened a new log
I20260929 16:36:03.375084 26316 ts_tablet_manager.cc:1397] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260929 16:36:03.375341 26314 consensus_queue.cc:260] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.377022 26314 ts_tablet_manager.cc:1428] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:36:03.384579 26315 raft_consensus.cc:348] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.385056 26315 raft_consensus.cc:374] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.385164 26315 raft_consensus.cc:729] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.385108 26316 raft_consensus.cc:348] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.385490 26316 raft_consensus.cc:374] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.385581 26316 raft_consensus.cc:729] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.385818 26315 consensus_queue.cc:260] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.386126 26316 consensus_queue.cc:260] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.387615 26315 ts_tablet_manager.cc:1428] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260929 16:36:03.387693 26316 ts_tablet_manager.cc:1428] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260929 16:36:03.458616 26012 tablet.cc:2369] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:36:03.478564 26151 tablet.cc:2369] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:36:03.505471 26290 tablet.cc:2369] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:36:03.549448 26321 raft_consensus.cc:482] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:03.549813 26321 raft_consensus.cc:504] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.551613 26321 leader_election.cc:288] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853)
I20260929 16:36:03.558655 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.551986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:44669 (local address 127.17.205.1:42977)
0929 16:36:03.552216 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.552221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.552732 (+ 511us) server_negotiation.cc:407] Connection header received
0929 16:36:03.553022 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.553028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.553120 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.553186 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.553950 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.554866 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.556870 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.557788 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.557908 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.557959 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.558299 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.558421 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:36:03.558683 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.551986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42977 (local address 127.17.205.3:44669)
0929 16:36:03.552643 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.552660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.552845 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.553393 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.553398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.553423 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.553813 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.553827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.554983 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.554989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.556730 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.556745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.557978 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.557985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.558142 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.558176 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.558276 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.558408 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":347,"threads_started":1}
I20260929 16:36:03.559053 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.552993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45617 (local address 127.17.205.2:38853)
0929 16:36:03.553178 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.553184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.553757 (+ 573us) server_negotiation.cc:407] Connection header received
0929 16:36:03.553894 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.553899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.553985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.554058 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.554605 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.555556 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.557322 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.558199 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.558336 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.558400 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.558728 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.558847 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:36:03.559103 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.552791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.17.205.3:45617)
0929 16:36:03.553651 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.553668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.553778 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.554095 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.554099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.554124 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.554452 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.554462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.555677 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.555682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.557180 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.557195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.558393 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.558399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.558556 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.558585 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.558687 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.558811 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":538,"threads_started":1}
I20260929 16:36:03.559942 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdfa7edc9994dbc9af734eee30949e8" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" is_pre_election: true
I20260929 16:36:03.559945 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdfa7edc9994dbc9af734eee30949e8" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44" is_pre_election: true
I20260929 16:36:03.560377 25961 raft_consensus.cc:2454] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 0.
I20260929 16:36:03.560377 26104 raft_consensus.cc:2454] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 0.
I20260929 16:36:03.560938 26170 leader_election.cc:302] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ddc09ed1c1a49919fac8b6f9dc8b845, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:03.561268 26321 raft_consensus.cc:2781] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:03.561411 26321 raft_consensus.cc:482] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:03.561517 26321 raft_consensus.cc:3037] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.563629 26321 raft_consensus.cc:504] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.564177 26321 leader_election.cc:288] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 election: Requested vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853)
I20260929 16:36:03.564556 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdfa7edc9994dbc9af734eee30949e8" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
I20260929 16:36:03.564672 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdfa7edc9994dbc9af734eee30949e8" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44"
I20260929 16:36:03.564762 25961 raft_consensus.cc:3037] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.564860 26104 raft_consensus.cc:3037] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.567862 25961 raft_consensus.cc:2454] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 1.
I20260929 16:36:03.567862 26104 raft_consensus.cc:2454] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 1.
I20260929 16:36:03.568300 26170 leader_election.cc:302] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:03.568606 26321 raft_consensus.cc:2781] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:03.569101 26321 raft_consensus.cc:686] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 LEADER]: Becoming Leader. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Running, Role: LEADER
I20260929 16:36:03.569555 26321 consensus_queue.cc:237] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.574227 25803 catalog_manager.cc:5495] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } health_report { overall_health: HEALTHY } } }
I20260929 16:36:03.623144 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.617305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.17.205.3:37551)
0929 16:36:03.617460 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.617464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.617536 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:36:03.617733 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.617739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.617857 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.618010 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:03.618312 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.619405 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.620740 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.621377 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.621486 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.621873 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:03.622579 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:03.622710 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.622790 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.622871 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:36:03.632027 25961 raft_consensus.cc:1261] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Refusing update from remote peer 43fb868ea8c34e9c9ce265f604cbabf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:03.632027 26104 raft_consensus.cc:1261] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Refusing update from remote peer 43fb868ea8c34e9c9ce265f604cbabf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:03.633013 26325 consensus_queue.cc:1035] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:03.633386 26321 consensus_queue.cc:1035] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:03.643795 26328 mvcc.cc:205] Tried to move back new op lower bound from 7334706231825645568 to 7334706231584673792. Current Snapshot: MvccSnapshot[applied={T|T < 7334706231825645568}]
I20260929 16:36:03.659199 25804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:03.659516 25804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:03.663110 25804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Sending DeleteTablet for 3 replicas of tablet 5cdfa7edc9994dbc9af734eee30949e8
I20260929 16:36:03.664325 26219 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:03 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:43846
I20260929 16:36:03.664619 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:03 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56216
I20260929 16:36:03.665679 25803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:36:03.666446 25803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:03.665261 26080 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdfa7edc9994dbc9af734eee30949e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:03 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:59660
I20260929 16:36:03.676035 25940 tablet_service.cc:1467] Processing CreateTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c (DEFAULT_TABLE table=test-table [id=a1a31d89338249e78d985d7cb5486d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.676674 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8631564b2b8e4dcf880d4f3c718c107c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.680110 26218 tablet_service.cc:1467] Processing CreateTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c (DEFAULT_TABLE table=test-table [id=a1a31d89338249e78d985d7cb5486d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.680819 26218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8631564b2b8e4dcf880d4f3c718c107c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.684130 26079 tablet_service.cc:1467] Processing CreateTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c (DEFAULT_TABLE table=test-table [id=a1a31d89338249e78d985d7cb5486d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:03.684703 26079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8631564b2b8e4dcf880d4f3c718c107c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.695843 26314 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Bootstrap starting.
I20260929 16:36:03.697504 26314 tablet_bootstrap.cc:654] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.700944 26316 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Bootstrap starting.
I20260929 16:36:03.701886 26315 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Bootstrap starting.
I20260929 16:36:03.702589 26316 tablet_bootstrap.cc:654] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.703474 26315 tablet_bootstrap.cc:654] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.703949 26340 tablet_replica.cc:330] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: stopping tablet replica
I20260929 16:36:03.704552 26340 raft_consensus.cc:2229] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:03.705195 26340 raft_consensus.cc:2258] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:03.709082 26340 ts_tablet_manager.cc:1905] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:03.718832 26340 ts_tablet_manager.cc:1918] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:03.719170 26340 log.cc:1196] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/5cdfa7edc9994dbc9af734eee30949e8
I20260929 16:36:03.719885 26340 ts_tablet_manager.cc:1939] T 5cdfa7edc9994dbc9af734eee30949e8 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting consensus metadata
I20260929 16:36:03.723905 25788 catalog_manager.cc:4841] TS 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977): tablet 5cdfa7edc9994dbc9af734eee30949e8 (table test-table [id=d16ff597da504ca09019a75610bcb197]) successfully deleted
I20260929 16:36:03.735774 26341 tablet_replica.cc:330] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: stopping tablet replica
I20260929 16:36:03.736464 26341 raft_consensus.cc:2229] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:03.737116 26341 raft_consensus.cc:2258] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:03.738755 26314 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: No bootstrap required, opened a new log
I20260929 16:36:03.738991 26314 ts_tablet_manager.cc:1397] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.008s
I20260929 16:36:03.740365 26316 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: No bootstrap required, opened a new log
I20260929 16:36:03.740684 26316 ts_tablet_manager.cc:1397] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.001s
I20260929 16:36:03.741732 26316 raft_consensus.cc:348] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.742074 26316 raft_consensus.cc:374] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.742223 26316 raft_consensus.cc:729] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.742594 26316 consensus_queue.cc:260] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.746029 26316 ts_tablet_manager.cc:1428] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260929 16:36:03.746963 26314 raft_consensus.cc:348] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.747357 26314 raft_consensus.cc:374] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.747493 26314 raft_consensus.cc:729] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.747850 26314 consensus_queue.cc:260] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.751106 26314 ts_tablet_manager.cc:1428] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260929 16:36:03.754894 26341 ts_tablet_manager.cc:1905] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:03.761145 26339 tablet_replica.cc:330] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: stopping tablet replica
I20260929 16:36:03.762526 26339 raft_consensus.cc:2229] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:36:03.762662 26341 ts_tablet_manager.cc:1918] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:03.763001 26341 log.cc:1196] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/5cdfa7edc9994dbc9af734eee30949e8
I20260929 16:36:03.763417 26339 raft_consensus.cc:2258] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:03.763581 26341 ts_tablet_manager.cc:1939] T 5cdfa7edc9994dbc9af734eee30949e8 P 150bc1e7d32847dca146b8c5c994da44: Deleting consensus metadata
I20260929 16:36:03.764846 26315 tablet_bootstrap.cc:492] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: No bootstrap required, opened a new log
I20260929 16:36:03.765004 26315 ts_tablet_manager.cc:1397] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent bootstrapping tablet: real 0.063s user 0.023s sys 0.026s
I20260929 16:36:03.765511 26339 ts_tablet_manager.cc:1905] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:03.765712 25788 catalog_manager.cc:4841] TS 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853): tablet 5cdfa7edc9994dbc9af734eee30949e8 (table test-table [id=d16ff597da504ca09019a75610bcb197]) successfully deleted
I20260929 16:36:03.765849 26315 raft_consensus.cc:348] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.766069 26315 raft_consensus.cc:374] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.766173 26315 raft_consensus.cc:729] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.766494 26315 consensus_queue.cc:260] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.767139 26315 ts_tablet_manager.cc:1428] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:36:03.773758 26339 ts_tablet_manager.cc:1918] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:03.773924 26339 log.cc:1196] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/5cdfa7edc9994dbc9af734eee30949e8
I20260929 16:36:03.774451 26339 ts_tablet_manager.cc:1939] T 5cdfa7edc9994dbc9af734eee30949e8 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting consensus metadata
I20260929 16:36:03.775974 25790 catalog_manager.cc:4841] TS 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551): tablet 5cdfa7edc9994dbc9af734eee30949e8 (table test-table [id=d16ff597da504ca09019a75610bcb197]) successfully deleted
I20260929 16:36:03.820819 26337 raft_consensus.cc:482] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:03.821079 26337 raft_consensus.cc:504] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.821938 26337 leader_election.cc:288] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853)
I20260929 16:36:03.822603 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8631564b2b8e4dcf880d4f3c718c107c" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44" is_pre_election: true
I20260929 16:36:03.822623 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8631564b2b8e4dcf880d4f3c718c107c" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" is_pre_election: true
I20260929 16:36:03.822856 26104 raft_consensus.cc:2454] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 0.
I20260929 16:36:03.822880 25961 raft_consensus.cc:2454] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 0.
I20260929 16:36:03.823345 26170 leader_election.cc:302] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:03.823642 26337 raft_consensus.cc:2781] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:03.823768 26337 raft_consensus.cc:482] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:03.823866 26337 raft_consensus.cc:3037] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.826031 26337 raft_consensus.cc:504] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.826610 26337 leader_election.cc:288] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 election: Requested vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853)
I20260929 16:36:03.827230 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8631564b2b8e4dcf880d4f3c718c107c" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44"
I20260929 16:36:03.827240 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8631564b2b8e4dcf880d4f3c718c107c" candidate_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
I20260929 16:36:03.827426 26104 raft_consensus.cc:3037] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.827471 25961 raft_consensus.cc:3037] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.829385 26104 raft_consensus.cc:2454] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 1.
I20260929 16:36:03.829789 26170 leader_election.cc:302] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:03.830077 26337 raft_consensus.cc:2781] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:03.830238 26337 raft_consensus.cc:686] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 LEADER]: Becoming Leader. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Running, Role: LEADER
I20260929 16:36:03.830314 25961 raft_consensus.cc:2454] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fb868ea8c34e9c9ce265f604cbabf6 in term 1.
I20260929 16:36:03.830530 26337 consensus_queue.cc:237] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.833254 25804 catalog_manager.cc:5495] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } health_report { overall_health: HEALTHY } } }
Sep 29 16:36:03 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699760, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.62@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:03.896828 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:36151 (local address 127.0.0.1:37932)
0929 16:36:03.882441 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.882458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.882591 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.883016 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.883020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.883770 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:03.884010 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.884026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.885486 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.885490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.886472 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.886482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.887444 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.887448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.887584 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.888343 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:03.888369 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:03.890748 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:03.894405 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:03.894416 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:03.894423 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:03.894821 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:03.895240 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:03.895245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:03.895249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:03.895400 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:03.896004 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:03.896011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:03.896182 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.896315 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.896538 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":549,"threads_started":1}
I20260929 16:36:03.897121 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.881639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.17.205.62:36151)
0929 16:36:03.882272 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.882279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.882540 (+ 261us) server_negotiation.cc:407] Connection header received
0929 16:36:03.882692 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.882697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.882813 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.882931 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:03.884175 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.885334 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.886607 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.887298 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.887424 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.890957 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:03.890964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:03.890970 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:03.891006 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:03.894241 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:03.894935 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:03.894940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:03.894944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:03.895085 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:03.895506 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:03.895511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:03.895514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:03.895872 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:03.896014 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.896430 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.896710 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":471,"spinlock_wait_cycles":1792,"thread_start_us":317,"threads_started":1}
I20260929 16:36:03.902020 25804 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:37932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:36:03.903147 25804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:03.910301 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 8848747ed88846d7ac1330357e525933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34241b246ea94989bc73c37f11be3e0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:36:03.910619 26219 tablet_service.cc:1467] Processing CreateTablet for tablet 8848747ed88846d7ac1330357e525933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34241b246ea94989bc73c37f11be3e0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:36:03.910984 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8848747ed88846d7ac1330357e525933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.911243 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8848747ed88846d7ac1330357e525933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.916864 26080 tablet_service.cc:1467] Processing CreateTablet for tablet 8848747ed88846d7ac1330357e525933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34241b246ea94989bc73c37f11be3e0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:36:03.917474 26080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8848747ed88846d7ac1330357e525933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:03.918556 26316 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Bootstrap starting.
I20260929 16:36:03.918893 26315 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: Bootstrap starting.
I20260929 16:36:03.920403 26316 tablet_bootstrap.cc:654] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.920848 26315 tablet_bootstrap.cc:654] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.925681 26314 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: Bootstrap starting.
I20260929 16:36:03.927577 26314 tablet_bootstrap.cc:654] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:03.928349 26315 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: No bootstrap required, opened a new log
I20260929 16:36:03.928574 26315 ts_tablet_manager.cc:1397] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 16:36:03.929528 26315 raft_consensus.cc:348] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.929860 26315 raft_consensus.cc:374] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.929953 26315 raft_consensus.cc:729] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.930305 26315 consensus_queue.cc:260] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.930832 26337 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:03.935254 26337 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:03.935952 26315 ts_tablet_manager.cc:1428] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 16:36:03.936558 26316 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: No bootstrap required, opened a new log
I20260929 16:36:03.936810 26316 ts_tablet_manager.cc:1397] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.009s
I20260929 16:36:03.937589 26316 raft_consensus.cc:348] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.937913 26316 raft_consensus.cc:374] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.938024 26316 raft_consensus.cc:729] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.938323 26316 consensus_queue.cc:260] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.939003 26316 ts_tablet_manager.cc:1428] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:36:03.938908 26322 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:03.939913 26322 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:03.941265 26314 tablet_bootstrap.cc:492] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: No bootstrap required, opened a new log
I20260929 16:36:03.941452 26314 ts_tablet_manager.cc:1397] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260929 16:36:03.942104 26314 raft_consensus.cc:348] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.942296 26314 raft_consensus.cc:374] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:03.942377 26314 raft_consensus.cc:729] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Initialized, Role: FOLLOWER
I20260929 16:36:03.942620 26314 consensus_queue.cc:260] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.943288 26314 ts_tablet_manager.cc:1428] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:36:03.943048 26320 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:03.943465 26320 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:03.945900 26320 raft_consensus.cc:482] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:03.946094 26320 raft_consensus.cc:504] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.949132 26320 leader_election.cc:288] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:03.957801 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.950022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:37949 (local address 127.17.205.1:42977)
0929 16:36:03.950571 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.950578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.950597 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:36:03.950663 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.950667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.950724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.950810 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.952548 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.953535 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.955889 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.956752 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.956929 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.957017 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.957365 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.957498 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421}
I20260929 16:36:03.957904 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.949369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42977 (local address 127.17.205.2:37949)
0929 16:36:03.950024 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.950044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.950184 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.950844 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.950850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.950886 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.951763 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.951780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.953729 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.953735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.955731 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.955744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.956989 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.956995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.957197 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.957236 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.957330 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.957472 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":40,"thread_start_us":1782,"threads_started":1}
I20260929 16:36:03.958626 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8848747ed88846d7ac1330357e525933" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" is_pre_election: true
I20260929 16:36:03.958938 25961 raft_consensus.cc:2454] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 0.
I20260929 16:36:03.959512 26031 leader_election.cc:302] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 2ddc09ed1c1a49919fac8b6f9dc8b845; no voters:
I20260929 16:36:03.960145 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.950338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57531 (local address 127.17.205.3:37551)
0929 16:36:03.950533 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.950540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.952347 (+ 1807us) server_negotiation.cc:407] Connection header received
0929 16:36:03.952513 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.952523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.952624 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.952724 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:03.953352 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.954531 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.957016 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.958217 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.958344 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.958411 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.959574 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.959790 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:36:03.960183 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.949530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37551 (local address 127.17.205.2:57531)
0929 16:36:03.952244 (+ 2714us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.952260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.952381 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.952766 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.952771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.952801 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:03.953164 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.953176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.954672 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.954682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.956825 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.956839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.959046 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.959122 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.959340 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.959377 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.959530 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.959689 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2616,"mutex_wait_us":30,"thread_start_us":2468,"threads_started":1}
I20260929 16:36:03.960409 26320 raft_consensus.cc:2781] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:03.960676 26320 raft_consensus.cc:482] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:03.960820 26320 raft_consensus.cc:3037] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.961671 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8848747ed88846d7ac1330357e525933" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" is_pre_election: true
I20260929 16:36:03.962159 26239 raft_consensus.cc:2454] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 0.
I20260929 16:36:03.963573 26320 raft_consensus.cc:504] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.964259 26320 leader_election.cc:288] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 election: Requested vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:03.964742 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8848747ed88846d7ac1330357e525933" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
I20260929 16:36:03.965004 25961 raft_consensus.cc:3037] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.965184 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8848747ed88846d7ac1330357e525933" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
I20260929 16:36:03.965394 26239 raft_consensus.cc:3037] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:03.967456 25961 raft_consensus.cc:2454] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 1.
I20260929 16:36:03.968291 26031 leader_election.cc:302] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 2ddc09ed1c1a49919fac8b6f9dc8b845; no voters:
I20260929 16:36:03.968681 26239 raft_consensus.cc:2454] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 1.
I20260929 16:36:03.968806 26320 raft_consensus.cc:2781] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:03.969621 26320 raft_consensus.cc:686] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [term 1 LEADER]: Becoming Leader. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Running, Role: LEADER
I20260929 16:36:03.970283 26320 consensus_queue.cc:237] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:03.972388 26358 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: TxnStatusTablet state changed. Reason: New leader 150bc1e7d32847dca146b8c5c994da44. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:03.972818 26358 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:36:03.974025 26360 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:36:03.974275 26360 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:36:03.974397 25804 catalog_manager.cc:5495] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2). New cstate: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } health_report { overall_health: UNKNOWN } } }
I20260929 16:36:03.995784 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.988429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.17.205.2:38853)
0929 16:36:03.988608 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:03.988615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:03.988647 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:36:03.988908 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:03.988913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:03.989012 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:03.989163 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:03.989596 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.990770 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.992763 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.993429 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.993570 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.994168 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:03.994940 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:03.995091 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:03.995322 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:36:03.995420 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:36:03.995803 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:03.987980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.0.0.1:59674)
0929 16:36:03.988493 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:03.988517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:03.988670 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:03.989203 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:03.989208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:03.989223 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:36:03.989428 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.989440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.990959 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.990966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.992597 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:03.992614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:03.993602 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:03.993609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:03.993846 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:03.993931 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:36:03.995109 (+ 1178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:36:03.995136 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:36:03.995276 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:36:03.995445 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334}
I20260929 16:36:04.000952 25961 raft_consensus.cc:1261] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Refusing update from remote peer 150bc1e7d32847dca146b8c5c994da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.001189 26239 raft_consensus.cc:1261] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Refusing update from remote peer 150bc1e7d32847dca146b8c5c994da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.001878 26358 consensus_queue.cc:1035] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:04.002642 26320 consensus_queue.cc:1035] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:04.012092 26322 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: TxnStatusTablet state changed. Reason: New leader 150bc1e7d32847dca146b8c5c994da44. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.012411 26322 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:04.014935 26320 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.015252 26320 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:36:04.015451 26330 mvcc.cc:205] Tried to move back new op lower bound from 7334706233338990592 to 7334706233226289152. Current Snapshot: MvccSnapshot[applied={T|T < 7334706233338990592}]
I20260929 16:36:04.018613 26329 mvcc.cc:205] Tried to move back new op lower bound from 7334706233338990592 to 7334706233226289152. Current Snapshot: MvccSnapshot[applied={T|T < 7334706233338990592}]
I20260929 16:36:04.020686 26337 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: TxnStatusTablet state changed. Reason: New leader 150bc1e7d32847dca146b8c5c994da44. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.021102 26337 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:04.023485 26320 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.023770 26320 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 150bc1e7d32847dca146b8c5c994da44: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:36:04.025913 26322 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.026274 26322 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 2ddc09ed1c1a49919fac8b6f9dc8b845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:04.034302 26337 tablet_replica.cc:439] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } } }
I20260929 16:36:04.034658 26337 tablet_replica.cc:442] T 8848747ed88846d7ac1330357e525933 P 43fb868ea8c34e9c9ce265f604cbabf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:36:04.056267 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.048060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:44811 (local address 127.17.205.2:38853)
0929 16:36:04.048511 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.048518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.048957 (+ 439us) server_negotiation.cc:407] Connection header received
0929 16:36:04.049235 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.049241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.049331 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.049426 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:04.050141 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.051368 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.053719 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.054420 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.054643 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.055264 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:04.055411 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:04.055516 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.055782 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.055934 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260929 16:36:04.056460 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.048055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.17.205.3:44811)
0929 16:36:04.048831 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:04.048853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:04.049044 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:04.049459 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:04.049465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:04.049480 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:36:04.049936 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.049952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.051522 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.051529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.053528 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.053548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.054645 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.054654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.054905 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.054998 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:36:04.055569 (+ 571us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:36:04.055591 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:36:04.055736 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:36:04.055945 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":463,"threads_started":1}
I20260929 16:36:04.069882 26104 raft_consensus.cc:1261] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Refusing update from remote peer 43fb868ea8c34e9c9ce265f604cbabf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.069890 25961 raft_consensus.cc:1261] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Refusing update from remote peer 43fb868ea8c34e9c9ce265f604cbabf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.070626 26337 consensus_queue.cc:1035] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:04.071199 26337 consensus_queue.cc:1035] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260929 16:36:04.128589 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.119672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:36151 (local address 127.17.205.3:37551)
0929 16:36:04.119860 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.119866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.120016 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:36:04.120286 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.120292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.120400 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.120463 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:04.122740 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.123989 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.126158 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.126910 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.127024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.127788 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:04.127930 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:04.128025 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.128181 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.128265 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:36:04.129016 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.119166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37551 (local address 127.17.205.2:36151)
0929 16:36:04.119920 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:04.119945 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:04.120132 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:04.120677 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:04.120681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:04.120698 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:36:04.122511 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.122527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.124137 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.124143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.125984 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.125999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.127288 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.127296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.127491 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.127568 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:36:04.128038 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:36:04.128050 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:36:04.128135 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:36:04.128291 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":534,"threads_started":1}
I20260929 16:36:04.184768 25803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:04.185053 25803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:04.188864 25803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Sending DeleteTablet for 3 replicas of tablet 8631564b2b8e4dcf880d4f3c718c107c
I20260929 16:36:04.190127 26080 tablet_service.cc:1514] Processing DeleteTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:59660
I20260929 16:36:04.190088 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56216
I20260929 16:36:04.190146 26219 tablet_service.cc:1514] Processing DeleteTablet for tablet 8631564b2b8e4dcf880d4f3c718c107c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:43846
I20260929 16:36:04.190407 26340 tablet_replica.cc:330] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: stopping tablet replica
I20260929 16:36:04.190426 26339 tablet_replica.cc:330] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: stopping tablet replica
I20260929 16:36:04.190446 26341 tablet_replica.cc:330] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: stopping tablet replica
I20260929 16:36:04.190605 26340 raft_consensus.cc:2229] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:04.190622 26339 raft_consensus.cc:2229] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:36:04.190634 26341 raft_consensus.cc:2229] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:04.190799 26340 raft_consensus.cc:2258] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.190850 26341 raft_consensus.cc:2258] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.190982 26339 raft_consensus.cc:2258] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.192323 26340 ts_tablet_manager.cc:1905] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.192822 26341 ts_tablet_manager.cc:1905] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.199736 26340 ts_tablet_manager.cc:1918] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:04.199918 26340 log.cc:1196] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/8631564b2b8e4dcf880d4f3c718c107c
I20260929 16:36:04.200011 26339 ts_tablet_manager.cc:1905] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.200309 26340 ts_tablet_manager.cc:1939] T 8631564b2b8e4dcf880d4f3c718c107c P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting consensus metadata
I20260929 16:36:04.201489 25788 catalog_manager.cc:4841] TS 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977): tablet 8631564b2b8e4dcf880d4f3c718c107c (table test-table [id=a1a31d89338249e78d985d7cb5486d68]) successfully deleted
I20260929 16:36:04.204957 26341 ts_tablet_manager.cc:1918] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:04.205248 26341 log.cc:1196] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/8631564b2b8e4dcf880d4f3c718c107c
I20260929 16:36:04.205737 26341 ts_tablet_manager.cc:1939] T 8631564b2b8e4dcf880d4f3c718c107c P 150bc1e7d32847dca146b8c5c994da44: Deleting consensus metadata
I20260929 16:36:04.206964 25788 catalog_manager.cc:4841] TS 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853): tablet 8631564b2b8e4dcf880d4f3c718c107c (table test-table [id=a1a31d89338249e78d985d7cb5486d68]) successfully deleted
I20260929 16:36:04.208206 26339 ts_tablet_manager.cc:1918] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:04.208429 26339 log.cc:1196] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/8631564b2b8e4dcf880d4f3c718c107c
I20260929 16:36:04.208833 26339 ts_tablet_manager.cc:1939] T 8631564b2b8e4dcf880d4f3c718c107c P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting consensus metadata
I20260929 16:36:04.209831 25790 catalog_manager.cc:4841] TS 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551): tablet 8631564b2b8e4dcf880d4f3c718c107c (table test-table [id=a1a31d89338249e78d985d7cb5486d68]) successfully deleted
Sep 29 16:36:04 dist-test-slave-3h01 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:36:04 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:04 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:36:04.256028 26352 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:04.244879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.17.205.62:36151)
0929 16:36:04.245053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.245059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.245445 (+ 386us) server_negotiation.cc:407] Connection header received
0929 16:36:04.245614 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.245618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.245684 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.245776 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:04.247073 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.247857 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.249121 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.249559 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.249690 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.255756 (+ 6066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37934: BlockingRecv error: recv got EOF from 127.0.0.1:37934 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 16:36:04 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM
I20260929 16:36:04.288627 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.268968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.17.205.62:36151)
0929 16:36:04.269137 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.269143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.271505 (+ 2362us) server_negotiation.cc:407] Connection header received
0929 16:36:04.271667 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.271672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.271755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.271855 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:04.273417 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.274460 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.276383 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.277002 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.277135 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.281874 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:04.281881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:04.281887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:04.281936 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:04.285311 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:04.286294 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:04.286301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:04.286305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:04.286472 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:04.286912 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:04.286917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:04.286921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:04.287346 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:04.287601 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.288011 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.288267 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:36:04.292356 25803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:36:04.293062 25803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:04.300387 25941 tablet_service.cc:1467] Processing CreateTablet for tablet cd2f21b8b375498d990613904c3236ca (DEFAULT_TABLE table=test-table [id=28701f25b7fe495f9b4ef42cf523677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.300482 26219 tablet_service.cc:1467] Processing CreateTablet for tablet cd2f21b8b375498d990613904c3236ca (DEFAULT_TABLE table=test-table [id=28701f25b7fe495f9b4ef42cf523677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.300894 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2f21b8b375498d990613904c3236ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.300894 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2f21b8b375498d990613904c3236ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.307425 26316 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Bootstrap starting.
I20260929 16:36:04.307173 26315 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Bootstrap starting.
I20260929 16:36:04.308089 26080 tablet_service.cc:1467] Processing CreateTablet for tablet cd2f21b8b375498d990613904c3236ca (DEFAULT_TABLE table=test-table [id=28701f25b7fe495f9b4ef42cf523677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.308553 26080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2f21b8b375498d990613904c3236ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.309075 26316 tablet_bootstrap.cc:654] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.309237 26315 tablet_bootstrap.cc:654] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.315351 26314 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Bootstrap starting.
I20260929 16:36:04.316932 26315 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: No bootstrap required, opened a new log
I20260929 16:36:04.316932 26316 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: No bootstrap required, opened a new log
I20260929 16:36:04.317025 26314 tablet_bootstrap.cc:654] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.317081 26315 ts_tablet_manager.cc:1397] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 16:36:04.317090 26316 ts_tablet_manager.cc:1397] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260929 16:36:04.317847 26315 raft_consensus.cc:348] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.318089 26315 raft_consensus.cc:374] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.318183 26315 raft_consensus.cc:729] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.318646 26315 consensus_queue.cc:260] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.318939 26316 raft_consensus.cc:348] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.319236 26316 raft_consensus.cc:374] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.319361 26316 raft_consensus.cc:729] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.319607 26316 consensus_queue.cc:260] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.320663 26316 ts_tablet_manager.cc:1428] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:04.321903 26315 ts_tablet_manager.cc:1428] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:36:04.328749 26314 tablet_bootstrap.cc:492] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: No bootstrap required, opened a new log
I20260929 16:36:04.329007 26314 ts_tablet_manager.cc:1397] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260929 16:36:04.329715 26314 raft_consensus.cc:348] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.329924 26314 raft_consensus.cc:374] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.330008 26314 raft_consensus.cc:729] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.330235 26314 consensus_queue.cc:260] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.330734 26314 ts_tablet_manager.cc:1428] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:36:04.523382 26320 raft_consensus.cc:482] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:04.523581 26320 raft_consensus.cc:504] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.524298 26320 leader_election.cc:288] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:04.524806 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2f21b8b375498d990613904c3236ca" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" is_pre_election: true
I20260929 16:36:04.524840 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2f21b8b375498d990613904c3236ca" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" is_pre_election: true
I20260929 16:36:04.525032 26239 raft_consensus.cc:2454] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 0.
I20260929 16:36:04.525027 25961 raft_consensus.cc:2454] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 0.
I20260929 16:36:04.525456 26031 leader_election.cc:302] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 2ddc09ed1c1a49919fac8b6f9dc8b845, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:04.525724 26320 raft_consensus.cc:2781] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:04.525823 26320 raft_consensus.cc:482] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:04.525911 26320 raft_consensus.cc:3037] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.527866 26320 raft_consensus.cc:504] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.528347 26320 leader_election.cc:288] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 election: Requested vote from peers 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:04.528771 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2f21b8b375498d990613904c3236ca" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845"
I20260929 16:36:04.528823 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2f21b8b375498d990613904c3236ca" candidate_uuid: "150bc1e7d32847dca146b8c5c994da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
I20260929 16:36:04.528949 25961 raft_consensus.cc:3037] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.528997 26239 raft_consensus.cc:3037] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.530941 26239 raft_consensus.cc:2454] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 1.
I20260929 16:36:04.531111 25961 raft_consensus.cc:2454] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 150bc1e7d32847dca146b8c5c994da44 in term 1.
I20260929 16:36:04.531379 26033 leader_election.cc:302] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 43fb868ea8c34e9c9ce265f604cbabf6; no voters:
I20260929 16:36:04.531728 26320 raft_consensus.cc:2781] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:04.531885 26320 raft_consensus.cc:686] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 1 LEADER]: Becoming Leader. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Running, Role: LEADER
I20260929 16:36:04.532171 26320 consensus_queue.cc:237] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.534657 25803 catalog_manager.cc:5495] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2). New cstate: current_term: 1 leader_uuid: "150bc1e7d32847dca146b8c5c994da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } health_report { overall_health: UNKNOWN } } }
I20260929 16:36:04.558348 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.553703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.17.205.2:38853)
0929 16:36:04.553891 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.553897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.553982 (+ 85us) server_negotiation.cc:407] Connection header received
0929 16:36:04.554200 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.554204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.554270 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.554340 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:04.554685 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.555394 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.556691 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.557166 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.557277 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.557697 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:04.557798 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:04.557891 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.557999 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.558077 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:36:04.561209 26239 raft_consensus.cc:1261] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Refusing update from remote peer 150bc1e7d32847dca146b8c5c994da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.561237 25961 raft_consensus.cc:1261] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Refusing update from remote peer 150bc1e7d32847dca146b8c5c994da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:04.561748 26320 consensus_queue.cc:1035] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:04.562005 26365 consensus_queue.cc:1035] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:04.576398 25802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:04.576601 25802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:04.579663 25802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Sending DeleteTablet for 3 replicas of tablet cd2f21b8b375498d990613904c3236ca
I20260929 16:36:04.580528 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2f21b8b375498d990613904c3236ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56216
I20260929 16:36:04.580574 26080 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2f21b8b375498d990613904c3236ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:59660
I20260929 16:36:04.580914 26341 tablet_replica.cc:330] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: stopping tablet replica
I20260929 16:36:04.580935 26340 tablet_replica.cc:330] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: stopping tablet replica
I20260929 16:36:04.581103 26341 raft_consensus.cc:2229] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:36:04.581096 26340 raft_consensus.cc:2229] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:04.581341 26340 raft_consensus.cc:2258] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.581647 26341 raft_consensus.cc:2258] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.581641 26219 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2f21b8b375498d990613904c3236ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:04 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:43846
I20260929 16:36:04.581918 26339 tablet_replica.cc:330] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: stopping tablet replica
I20260929 16:36:04.582084 26339 raft_consensus.cc:2229] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:04.582321 26339 raft_consensus.cc:2258] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:04.582536 26341 ts_tablet_manager.cc:1905] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.583431 26340 ts_tablet_manager.cc:1905] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.584113 25802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:36:04.584803 25802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:04.589640 26340 ts_tablet_manager.cc:1918] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:04.589653 26341 ts_tablet_manager.cc:1918] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:04.589789 26341 log.cc:1196] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/cd2f21b8b375498d990613904c3236ca
I20260929 16:36:04.589990 26340 log.cc:1196] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/cd2f21b8b375498d990613904c3236ca
I20260929 16:36:04.590248 26341 ts_tablet_manager.cc:1939] T cd2f21b8b375498d990613904c3236ca P 150bc1e7d32847dca146b8c5c994da44: Deleting consensus metadata
I20260929 16:36:04.590500 26340 ts_tablet_manager.cc:1939] T cd2f21b8b375498d990613904c3236ca P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting consensus metadata
I20260929 16:36:04.591531 25788 catalog_manager.cc:4841] TS 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853): tablet cd2f21b8b375498d990613904c3236ca (table test-table [id=28701f25b7fe495f9b4ef42cf523677d]) successfully deleted
I20260929 16:36:04.592496 26080 tablet_service.cc:1467] Processing CreateTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 (DEFAULT_TABLE table=test-table [id=757e29a84b8149ac8a0ea9221283e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.592738 26339 ts_tablet_manager.cc:1905] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:04.592792 26219 tablet_service.cc:1467] Processing CreateTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 (DEFAULT_TABLE table=test-table [id=757e29a84b8149ac8a0ea9221283e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.592929 26080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9810dab5d34b04bfc6b082ef985f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.593187 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9810dab5d34b04bfc6b082ef985f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.599689 25941 tablet_service.cc:1467] Processing CreateTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 (DEFAULT_TABLE table=test-table [id=757e29a84b8149ac8a0ea9221283e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:04.600023 26314 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Bootstrap starting.
I20260929 16:36:04.600203 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9810dab5d34b04bfc6b082ef985f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:04.600201 26339 ts_tablet_manager.cc:1918] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:04.600339 26339 log.cc:1196] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/cd2f21b8b375498d990613904c3236ca
I20260929 16:36:04.600781 26339 ts_tablet_manager.cc:1939] T cd2f21b8b375498d990613904c3236ca P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting consensus metadata
I20260929 16:36:04.601987 26314 tablet_bootstrap.cc:654] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.602080 25790 catalog_manager.cc:4841] TS 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551): tablet cd2f21b8b375498d990613904c3236ca (table test-table [id=28701f25b7fe495f9b4ef42cf523677d]) successfully deleted
I20260929 16:36:04.603355 25788 catalog_manager.cc:4841] TS 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977): tablet cd2f21b8b375498d990613904c3236ca (table test-table [id=28701f25b7fe495f9b4ef42cf523677d]) successfully deleted
I20260929 16:36:04.604786 26314 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: No bootstrap required, opened a new log
I20260929 16:36:04.604969 26314 ts_tablet_manager.cc:1397] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 16:36:04.605664 26314 raft_consensus.cc:348] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.605973 26314 raft_consensus.cc:374] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.606066 26314 raft_consensus.cc:729] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bc1e7d32847dca146b8c5c994da44, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.606338 26314 consensus_queue.cc:260] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.606854 26314 ts_tablet_manager.cc:1428] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:36:04.608143 26315 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Bootstrap starting.
I20260929 16:36:04.608914 26316 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Bootstrap starting.
I20260929 16:36:04.609714 26315 tablet_bootstrap.cc:654] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.610435 26316 tablet_bootstrap.cc:654] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:04.611891 26315 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: No bootstrap required, opened a new log
I20260929 16:36:04.612039 26315 ts_tablet_manager.cc:1397] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 16:36:04.612869 26315 raft_consensus.cc:348] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.613144 26315 raft_consensus.cc:374] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.613227 26315 raft_consensus.cc:729] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb868ea8c34e9c9ce265f604cbabf6, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.613485 26315 consensus_queue.cc:260] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.613999 26315 ts_tablet_manager.cc:1428] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:36:04.624641 26316 tablet_bootstrap.cc:492] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: No bootstrap required, opened a new log
I20260929 16:36:04.624845 26316 ts_tablet_manager.cc:1397] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260929 16:36:04.625658 26316 raft_consensus.cc:348] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.625952 26316 raft_consensus.cc:374] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:04.626101 26316 raft_consensus.cc:729] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Initialized, Role: FOLLOWER
I20260929 16:36:04.626446 26316 consensus_queue.cc:260] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.628054 26316 ts_tablet_manager.cc:1428] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 16:36:04.966975 26322 raft_consensus.cc:482] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:04.967257 26322 raft_consensus.cc:504] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.968783 26322 leader_election.cc:288] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:04.976125 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.969284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:47661 (local address 127.17.205.2:38853)
0929 16:36:04.969524 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.969531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.970535 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 16:36:04.970839 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.970845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.970925 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.971011 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:04.971603 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.972510 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.974387 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.975196 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.975367 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.975442 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.975718 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.975862 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:36:04.976220 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.968984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.17.205.1:47661)
0929 16:36:04.970396 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:04.970422 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:04.970662 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:04.971041 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:04.971046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:04.971073 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:04.971412 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.971425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.972638 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.972644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.974200 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.974212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.975380 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.975386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.975575 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.975601 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:36:04.975672 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:36:04.975784 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":21,"thread_start_us":1508,"threads_started":1}
I20260929 16:36:04.976264 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.969099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37551 (local address 127.17.205.1:59377)
0929 16:36:04.970777 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:04.970793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:04.970918 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:04.971333 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:04.971336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:04.971356 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:04.971720 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.971732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.972841 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.972844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.974364 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:04.974378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.975523 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.975527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:04.975650 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.975681 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:36:04.975769 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:36:04.975888 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":1,"thread_start_us":976,"threads_started":1}
I20260929 16:36:04.976207 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:04.969455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:59377 (local address 127.17.205.3:37551)
0929 16:36:04.970514 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:04.970521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:04.970885 (+ 364us) server_negotiation.cc:407] Connection header received
0929 16:36:04.971068 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:04.971073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:04.971148 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:04.971297 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:04.971866 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.972723 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.974513 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:04.975305 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:04.975459 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:04.975533 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:04.975804 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:36:04.975946 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":821,"threads_started":1}
I20260929 16:36:04.976953 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9810dab5d34b04bfc6b082ef985f50" candidate_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44" is_pre_election: true
I20260929 16:36:04.977277 26101 raft_consensus.cc:2454] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ddc09ed1c1a49919fac8b6f9dc8b845 in term 0.
I20260929 16:36:04.977303 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9810dab5d34b04bfc6b082ef985f50" candidate_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" is_pre_election: true
I20260929 16:36:04.977509 26239 raft_consensus.cc:2454] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ddc09ed1c1a49919fac8b6f9dc8b845 in term 0.
I20260929 16:36:04.977856 25891 leader_election.cc:302] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 2ddc09ed1c1a49919fac8b6f9dc8b845; no voters:
I20260929 16:36:04.978296 26322 raft_consensus.cc:2781] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:04.978416 26322 raft_consensus.cc:482] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:04.978492 26322 raft_consensus.cc:3037] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.980676 26322 raft_consensus.cc:504] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.981384 26322 leader_election.cc:288] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [CANDIDATE]: Term 1 election: Requested vote from peers 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853), 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551)
I20260929 16:36:04.981894 26239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9810dab5d34b04bfc6b082ef985f50" candidate_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fb868ea8c34e9c9ce265f604cbabf6"
I20260929 16:36:04.981910 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9810dab5d34b04bfc6b082ef985f50" candidate_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "150bc1e7d32847dca146b8c5c994da44"
I20260929 16:36:04.982105 26239 raft_consensus.cc:3037] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.982106 26101 raft_consensus.cc:3037] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:04.984234 26101 raft_consensus.cc:2454] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ddc09ed1c1a49919fac8b6f9dc8b845 in term 1.
I20260929 16:36:04.984463 26239 raft_consensus.cc:2454] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ddc09ed1c1a49919fac8b6f9dc8b845 in term 1.
I20260929 16:36:04.984669 25891 leader_election.cc:302] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 150bc1e7d32847dca146b8c5c994da44, 2ddc09ed1c1a49919fac8b6f9dc8b845; no voters:
I20260929 16:36:04.985000 26322 raft_consensus.cc:2781] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:04.985426 26322 raft_consensus.cc:686] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 LEADER]: Becoming Leader. State: Replica: 2ddc09ed1c1a49919fac8b6f9dc8b845, State: Running, Role: LEADER
I20260929 16:36:04.985802 26322 consensus_queue.cc:237] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } }
I20260929 16:36:04.989485 25804 catalog_manager.cc:5495] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddc09ed1c1a49919fac8b6f9dc8b845" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 } health_report { overall_health: UNKNOWN } } }
I20260929 16:36:05.092178 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.086184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.17.205.1:42977)
0929 16:36:05.086697 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.086703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.086776 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:36:05.087012 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.087018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.087141 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.087322 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:05.087763 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.088522 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.089816 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.090238 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.090343 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.090792 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:05.091529 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:05.091679 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.091754 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.091840 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260929 16:36:05.106608 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.101669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:48255 (local address 127.17.205.2:38853)
0929 16:36:05.101864 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.101869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.102405 (+ 536us) server_negotiation.cc:407] Connection header received
0929 16:36:05.102546 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.102550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.102615 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.102709 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:05.103179 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.103936 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.105245 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.105632 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.105724 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.106081 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:05.106194 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:05.106256 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.106345 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.106404 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:36:05.106691 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.101667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38853 (local address 127.17.205.1:48255)
0929 16:36:05.102303 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:05.102319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:05.102440 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:05.102748 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:05.102751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:05.102766 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:36:05.103014 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:05.103021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.104057 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.104061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:05.105114 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:05.105126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.105774 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.105778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:05.105901 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.105955 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:36:05.106264 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:36:05.106271 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:36:05.106323 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 16:36:05.106423 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260929 16:36:05.114567 26101 raft_consensus.cc:1261] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Refusing update from remote peer 2ddc09ed1c1a49919fac8b6f9dc8b845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:05.114599 26239 raft_consensus.cc:1261] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Refusing update from remote peer 2ddc09ed1c1a49919fac8b6f9dc8b845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:05.115265 26401 consensus_queue.cc:1035] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "150bc1e7d32847dca146b8c5c994da44" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:05.115594 26322 consensus_queue.cc:1035] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43fb868ea8c34e9c9ce265f604cbabf6" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:05.140516 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.135321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:40297 (local address 127.17.205.1:42977)
0929 16:36:05.135611 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.135617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.136063 (+ 446us) server_negotiation.cc:407] Connection header received
0929 16:36:05.136242 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.136248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.136325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.136409 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:05.136829 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.137513 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.138796 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.139360 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.139497 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.139991 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:05.140091 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:05.140163 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.140254 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.140318 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 16:36:05.140591 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.135307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42977 (local address 127.17.205.2:40297)
0929 16:36:05.135965 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:05.135988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:05.136106 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:05.136424 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:05.136429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:05.136441 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:36:05.136662 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:05.136675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.137625 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.137629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:05.138672 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:05.138684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.139568 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.139575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:05.139766 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.139815 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:36:05.140165 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:36:05.140174 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:36:05.140235 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 16:36:05.140335 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":376,"threads_started":1}
I20260929 16:36:05.171873 25804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:05.172118 25804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:05.174954 25804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f6f65826ee483285ee73db2c55eebd: Sending DeleteTablet for 3 replicas of tablet fb9810dab5d34b04bfc6b082ef985f50
I20260929 16:36:05.175823 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:05 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56216
I20260929 16:36:05.176065 26219 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:05 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:43846
I20260929 16:36:05.176584 26080 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9810dab5d34b04bfc6b082ef985f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:05 UTC) from {username='oryx', principal='oryx/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:59660
I20260929 16:36:05.177166 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 25875
I20260929 16:36:05.185380 26427 tablet_replica.cc:330] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: stopping tablet replica
I20260929 16:36:05.185644 26427 raft_consensus.cc:2229] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:05.185899 26427 raft_consensus.cc:2258] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:05.186075 26428 tablet_replica.cc:330] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: stopping tablet replica
I20260929 16:36:05.186321 26428 raft_consensus.cc:2229] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:05.186630 26428 raft_consensus.cc:2258] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:05.187687 26428 ts_tablet_manager.cc:1905] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:05.187702 26427 ts_tablet_manager.cc:1905] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:05.193116 26426 tablet_replica.cc:330] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: stopping tablet replica
I20260929 16:36:05.193346 26426 raft_consensus.cc:2229] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:36:05.193938 26426 raft_consensus.cc:2258] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:05.194222 26427 ts_tablet_manager.cc:1918] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:05.194356 26427 log.cc:1196] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/fb9810dab5d34b04bfc6b082ef985f50
I20260929 16:36:05.194706 26428 ts_tablet_manager.cc:1918] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:05.194787 26427 ts_tablet_manager.cc:1939] T fb9810dab5d34b04bfc6b082ef985f50 P 43fb868ea8c34e9c9ce265f604cbabf6: Deleting consensus metadata
I20260929 16:36:05.194962 26428 log.cc:1196] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/fb9810dab5d34b04bfc6b082ef985f50
I20260929 16:36:05.195485 26428 ts_tablet_manager.cc:1939] T fb9810dab5d34b04bfc6b082ef985f50 P 150bc1e7d32847dca146b8c5c994da44: Deleting consensus metadata
I20260929 16:36:05.195940 25790 catalog_manager.cc:4841] TS 43fb868ea8c34e9c9ce265f604cbabf6 (127.17.205.3:37551): tablet fb9810dab5d34b04bfc6b082ef985f50 (table test-table [id=757e29a84b8149ac8a0ea9221283e742]) successfully deleted
I20260929 16:36:05.196703 25788 catalog_manager.cc:4841] TS 150bc1e7d32847dca146b8c5c994da44 (127.17.205.2:38853): tablet fb9810dab5d34b04bfc6b082ef985f50 (table test-table [id=757e29a84b8149ac8a0ea9221283e742]) successfully deleted
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.1@KRBTEST.COM
I20260929 16:36:05.197306 26426 ts_tablet_manager.cc:1905] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:05.204634 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.178456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.17.205.1:42977)
0929 16:36:05.178673 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.178680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.185512 (+ 6832us) server_negotiation.cc:407] Connection header received
0929 16:36:05.185985 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.185991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.186078 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.186202 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:05.188466 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.189556 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.191335 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.191976 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.192078 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.198240 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:05.198252 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:05.198268 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:05.198309 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:05.201325 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.202013 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.202023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.202027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.202208 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.202656 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.202664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.202667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.203150 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:05.203476 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.204020 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.204262 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:36:05.204661 26426 ts_tablet_manager.cc:1918] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:36:05.204828 26426 log.cc:1196] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/fb9810dab5d34b04bfc6b082ef985f50
I20260929 16:36:05.205320 26426 ts_tablet_manager.cc:1939] T fb9810dab5d34b04bfc6b082ef985f50 P 2ddc09ed1c1a49919fac8b6f9dc8b845: Deleting consensus metadata
I20260929 16:36:05.206005 26006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:05.206382 25788 catalog_manager.cc:4841] TS 2ddc09ed1c1a49919fac8b6f9dc8b845 (127.17.205.1:42977): tablet fb9810dab5d34b04bfc6b082ef985f50 (table test-table [id=757e29a84b8149ac8a0ea9221283e742]) successfully deleted
I20260929 16:36:05.311465 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 25875
I20260929 16:36:05.327906 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26015
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.2@KRBTEST.COM
I20260929 16:36:05.343065 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.328657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.17.205.2:38853)
0929 16:36:05.328829 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.328833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.328850 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:36:05.328975 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.328978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.329044 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.329122 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:05.330200 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.330882 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.332218 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.332635 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.332739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.336674 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:05.336691 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:05.336707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:05.336748 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:05.339892 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.340842 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.340881 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.340884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.341086 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.341593 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.341598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.341601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.342062 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:05.342334 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.342557 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.342737 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:36:05.343735 26145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:05.484568 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26015
I20260929 16:36:05.505270 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26154
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.205.3@KRBTEST.COM
I20260929 16:36:05.520360 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:05.505951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.17.205.3:37551)
0929 16:36:05.506381 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:05.506385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:05.506400 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:36:05.506452 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:05.506455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:05.506511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:05.506596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:05.507842 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.508541 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.509842 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:05.510279 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:05.510380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:05.514459 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:05.514477 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:05.514494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:05.514540 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:05.517311 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.518041 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.518049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.518052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.518200 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:05.518617 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:05.518623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:05.518625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:05.519048 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:05.519456 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:05.519734 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:36:05.519972 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260929 16:36:05.521071 26284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:05.642679 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26154
I20260929 16:36:05.660351 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 25772
I20260929 16:36:05.661465 25844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:05.775971 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 25772
2026-09-29T16:36:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6799 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[26437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[26437](info): set up 2 sockets
Sep 29 16:36:05 dist-test-slave-3h01 krb5kdc[26437](info): commencing operation
krb5kdc: starting...
W20260929 16:36:06.775890 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:36:06 dist-test-slave-3h01 krb5kdc[26437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:36:06.992954 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34885
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:07.310846 26450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:07.311148 26450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:07.311276 26450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:07.318552 26450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:07.318681 26450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:07.318729 26450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:07.318785 26450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:07.318831 26450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:07.328702 26450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:07.329958 26450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:07.331257 26450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:07.342898 26458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:07.342931 26455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:07.342962 26456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:07.343631 26450 server_base.cc:1034] running on GCE node
I20260929 16:36:07.344460 26450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:07.346068 26450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:07.347318 26450 hybrid_clock.cc:648] HybridClock initialized: now 1790699767347277 us; error 41 us; skew 500 ppm
I20260929 16:36:07.347738 26450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:36:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1646 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:36:07 dist-test-slave-3h01 krb5kdc[26464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:36:07 dist-test-slave-3h01 krb5kdc[26464](info): set up 2 sockets
Sep 29 16:36:07 dist-test-slave-3h01 krb5kdc[26464](info): commencing operation
krb5kdc: starting...
W20260929 16:36:08.427995 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:36:08 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:36:08.643162 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42549
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:08.959677 26477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:08.960014 26477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:08.960146 26477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:08.967474 26477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:08.967613 26477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:08.967700 26477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:08.967773 26477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:08.967835 26477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:08.977777 26477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:42549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:08.979257 26477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:08.980719 26477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:08.992785 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:08.992820 26483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:08.992844 26485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:08.993700 26477 server_base.cc:1034] running on GCE node
I20260929 16:36:08.994555 26477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:08.996156 26477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:08.997390 26477 hybrid_clock.cc:648] HybridClock initialized: now 1790699768997326 us; error 44 us; skew 500 ppm
I20260929 16:36:08.997772 26477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:09 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699769, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:09.004593 26477 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:36:09.006455 26477 webserver.cc:460] Webserver started at http://127.17.205.62:32787/ using document root <none> and password file <none>
I20260929 16:36:09.007247 26477 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:09.007385 26477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:09.007709 26477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:09.010797 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "cf03a4115b0344b88146b6f452ca3d53"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "acc5019a5ea65341fc9bc58c078d42e0"
server_key_iv: "8112d54469912379df32bb424dbf9b4f"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.011647 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "cf03a4115b0344b88146b6f452ca3d53"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "acc5019a5ea65341fc9bc58c078d42e0"
server_key_iv: "8112d54469912379df32bb424dbf9b4f"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.017138 26477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:36:09.020903 26492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:09.022466 26477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:36:09.022642 26477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "cf03a4115b0344b88146b6f452ca3d53"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "acc5019a5ea65341fc9bc58c078d42e0"
server_key_iv: "8112d54469912379df32bb424dbf9b4f"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.022850 26477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:09.042125 26477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:09.046097 26477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:09.046368 26477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:09.064015 26477 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:42549
I20260929 16:36:09.064018 26543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:42549 every 8 connection(s)
I20260929 16:36:09.065441 26477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:09.069283 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:09.072750 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26477
I20260929 16:36:09.072926 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:36:09.073527 18228 external_mini_cluster.cc:1428] Setting key 86ef2bb0748c796bd6b1efa62da768ca
I20260929 16:36:09.082293 26544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Bootstrap starting.
I20260929 16:36:09.085872 26544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:09.086854 26544 log.cc:824] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:36:09 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:09.089898 26544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: No bootstrap required, opened a new log
I20260929 16:36:09.097162 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:09.075460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.17.205.62:42549)
0929 16:36:09.076219 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:09.076235 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:09.076285 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:36:09.077974 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:09.078007 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:09.078572 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:09.079044 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:09.080486 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.082433 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.083802 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.084852 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.085033 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:09.088766 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:09.088777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:09.088797 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:09.088843 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:09.092602 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:09.093328 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:09.093337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:09.093346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:09.093553 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:09.093894 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:09.093898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:09.093901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:09.094600 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:09.094907 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:09.095277 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:36:09.095761 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
I20260929 16:36:09.099593 26544 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:09.099959 26544 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:09.100070 26544 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf03a4115b0344b88146b6f452ca3d53, State: Initialized, Role: FOLLOWER
I20260929 16:36:09.100647 26544 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:09.100888 26544 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:09.101009 26544 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:09.101167 26544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:09.103233 26544 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:09.103715 26544 leader_election.cc:302] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf03a4115b0344b88146b6f452ca3d53; no voters:
I20260929 16:36:09.104280 26544 leader_election.cc:288] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:36:09.104426 26549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:09.105062 26549 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 LEADER]: Becoming Leader. State: Replica: cf03a4115b0344b88146b6f452ca3d53, State: Running, Role: LEADER
I20260929 16:36:09.105584 26549 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:09.106460 26544 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:09.108680 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf03a4115b0344b88146b6f452ca3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } } }
I20260929 16:36:09.108714 26551 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf03a4115b0344b88146b6f452ca3d53. Latest consensus state: current_term: 1 leader_uuid: "cf03a4115b0344b88146b6f452ca3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } } }
I20260929 16:36:09.109071 26551 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:09.109072 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:09.112524 26557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:09.116804 26557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:09.125152 26557 catalog_manager.cc:1347] Generated new cluster ID: cf438691d63f47a088200a9975fc97cd
I20260929 16:36:09.125276 26557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:09.139514 26557 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:36:09.141356 26557 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:09.150676 26557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Generated new TSK 0
I20260929 16:36:09.151746 26557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:36:09.200049 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:09.523370 26570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:09.523665 26570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:09.523794 26570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:09.531018 26570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:09.531131 26570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:09.531299 26570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:09.540918 26570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:09.542207 26570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:09.543596 26570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:09.556164 26575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:09.556471 26576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:09.556502 26578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:09.557315 26570 server_base.cc:1034] running on GCE node
I20260929 16:36:09.557885 26570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:09.558794 26570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:09.559988 26570 hybrid_clock.cc:648] HybridClock initialized: now 1790699769559940 us; error 45 us; skew 500 ppm
I20260929 16:36:09.560302 26570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:09 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699769, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:09.563292 26570 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:36:09.564780 26570 webserver.cc:460] Webserver started at http://127.17.205.1:34729/ using document root <none> and password file <none>
I20260929 16:36:09.565474 26570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:09.565585 26570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:09.565864 26570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:09.568712 26570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "db5dfccc2d4547dead919dfa7fb86580"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "dbf1153a9aea6d5d541cd4f3706be8e0"
server_key_iv: "78cddb79a5c16e9879ca20e5e1b9af37"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.569447 26570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "db5dfccc2d4547dead919dfa7fb86580"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "dbf1153a9aea6d5d541cd4f3706be8e0"
server_key_iv: "78cddb79a5c16e9879ca20e5e1b9af37"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.574422 26570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 16:36:09.577979 26585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:09.579438 26570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:36:09.579607 26570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "db5dfccc2d4547dead919dfa7fb86580"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "dbf1153a9aea6d5d541cd4f3706be8e0"
server_key_iv: "78cddb79a5c16e9879ca20e5e1b9af37"
server_key_version: "encryptionkey@0"
I20260929 16:36:09.579802 26570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:09.603181 26570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:09.607854 26570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:09.608157 26570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:09.609160 26570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:09.610607 26570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:09.610697 26570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:09.610822 26570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:09.610877 26570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:09.640801 26570 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:45201
I20260929 16:36:09.640877 26697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:45201 every 8 connection(s)
I20260929 16:36:09.642069 26570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:09.650609 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26570
I20260929 16:36:09.650777 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:36:09.651422 18228 external_mini_cluster.cc:1428] Setting key f1db3f10b0c047777e36fed95a41c2ca
Sep 29 16:36:09 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699769, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:09.661358 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:09.644494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:48653 (local address 127.17.205.62:42549)
0929 16:36:09.644904 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:09.644908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:09.645920 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 16:36:09.647753 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:09.647758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:09.647839 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:09.647931 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:09.649416 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.650190 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.651846 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.652237 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.652346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:09.656053 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:09.656058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:09.656061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:09.656093 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:09.658413 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:09.659124 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:09.659130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:09.659141 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:09.659297 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:09.659694 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:09.659699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:09.659703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:09.659969 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:09.660094 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:09.660769 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 16:36:09.660992 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260929 16:36:09.662313 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:09.645071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:48653)
0929 16:36:09.645761 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:09.645805 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:09.647475 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:09.648174 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:09.648182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:09.648698 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:09.649258 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:09.649274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.650322 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.650326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:09.651718 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:09.651729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:09.652443 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:09.652450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:09.652671 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:09.653360 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:09.653393 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:09.655838 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:09.658553 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:09.658562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:09.658581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:09.659003 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:09.659393 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:09.659399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:09.659403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:09.659584 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:09.660085 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:09.660092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:09.660429 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 16:36:09.660670 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 16:36:09.661211 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":255,"threads_started":1}
I20260929 16:36:09.663794 26698 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42549
I20260929 16:36:09.664103 26698 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:09.664913 26698 heartbeater.cc:511] Master 127.17.205.62:42549 requested a full tablet report, sending...
I20260929 16:36:09.667459 26509 ts_manager.cc:194] Registered new tserver with Master: db5dfccc2d4547dead919dfa7fb86580 (127.17.205.1:45201)
I20260929 16:36:09.669569 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:48653
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:36:09.696274 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:10.012274 26704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:10.012560 26704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:10.012652 26704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:10.019973 26704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:10.020103 26704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:10.020219 26704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:10.030124 26704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:10.031461 26704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:10.032747 26704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:10.046169 26712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:10.046200 26709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:10.046248 26710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:10.046835 26704 server_base.cc:1034] running on GCE node
I20260929 16:36:10.047600 26704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:10.048559 26704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:10.049768 26704 hybrid_clock.cc:648] HybridClock initialized: now 1790699770049713 us; error 50 us; skew 500 ppm
I20260929 16:36:10.050076 26704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699770, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:10.053201 26704 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:36:10.054705 26704 webserver.cc:460] Webserver started at http://127.17.205.2:44859/ using document root <none> and password file <none>
I20260929 16:36:10.055469 26704 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:10.055579 26704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:10.056002 26704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:10.059020 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "e556a951d2bd4ff590aa8a148b7c631d"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "70306b65f4a1c0c66c750ab4191aba95"
server_key_iv: "5850b66c5e878f02ceee15ec8abd5eb3"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.059902 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "e556a951d2bd4ff590aa8a148b7c631d"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "70306b65f4a1c0c66c750ab4191aba95"
server_key_iv: "5850b66c5e878f02ceee15ec8abd5eb3"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.065477 26704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:36:10.069111 26719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.070626 26704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:10.070788 26704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "e556a951d2bd4ff590aa8a148b7c631d"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "70306b65f4a1c0c66c750ab4191aba95"
server_key_iv: "5850b66c5e878f02ceee15ec8abd5eb3"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.070976 26704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:10.086464 26704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:10.090296 26704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:10.090559 26704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:10.091554 26704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:10.092844 26704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:10.092918 26704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.093040 26704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:10.093096 26704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.121963 26704 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38469
I20260929 16:36:10.122000 26831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38469 every 8 connection(s)
I20260929 16:36:10.123258 26704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:10.126154 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26704
I20260929 16:36:10.126320 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:36:10.126878 18228 external_mini_cluster.cc:1428] Setting key 5a1a414fde8beaec465f209e333090bf
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699770, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:10.142975 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.125780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33727 (local address 127.17.205.62:42549)
0929 16:36:10.125945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:10.125949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:10.127530 (+ 1581us) server_negotiation.cc:407] Connection header received
0929 16:36:10.129512 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:10.129518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:10.129599 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:10.129682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:10.131264 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.131978 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.133520 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.133920 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.134031 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.137562 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:10.137568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:10.137571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:10.137605 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:10.140062 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.140816 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.140821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.140823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.140959 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.141381 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.141384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.141386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.141649 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:10.141784 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:10.142568 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 16:36:10.142752 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:36:10.144560 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.126399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:33727)
0929 16:36:10.127352 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:10.127395 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:10.129239 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:10.129953 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:10.129962 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:10.130477 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:10.131074 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:10.131092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.132133 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.132137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:10.133378 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:10.133390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.134136 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.134143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:10.134371 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.134996 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:10.135019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:10.137330 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:10.140244 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:10.140254 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:10.140274 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:10.140701 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:10.141065 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:10.141070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:10.141074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:10.141279 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:10.141774 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:10.141783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:10.142142 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 16:36:10.142488 (+ 346us) client_negotiation.cc:240] Negotiation successful
0929 16:36:10.143062 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:36:10.146035 26832 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42549
I20260929 16:36:10.146338 26832 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:10.147096 26832 heartbeater.cc:511] Master 127.17.205.62:42549 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:36:10.148630 26509 ts_manager.cc:194] Registered new tserver with Master: e556a951d2bd4ff590aa8a148b7c631d (127.17.205.2:38469)
I20260929 16:36:10.149425 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:33727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:36:10.172353 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:10.486951 26837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:10.487269 26837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:10.487365 26837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:10.494444 26837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:10.494550 26837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:10.494673 26837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:10.504287 26837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:10.505519 26837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:10.506788 26837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:10.519248 26842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:10.519271 26845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:10.519188 26843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:10.520165 26837 server_base.cc:1034] running on GCE node
I20260929 16:36:10.520720 26837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:10.521662 26837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:10.522863 26837 hybrid_clock.cc:648] HybridClock initialized: now 1790699770522801 us; error 49 us; skew 500 ppm
I20260929 16:36:10.523187 26837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699770, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:10.526203 26837 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:36:10.527746 26837 webserver.cc:460] Webserver started at http://127.17.205.3:37153/ using document root <none> and password file <none>
I20260929 16:36:10.528420 26837 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:10.528529 26837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:10.528818 26837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:10.531762 26837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "197e7702e73a4993841d536cf7861d4a"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "b8a019711264d26eab503e6d78d73b48"
server_key_iv: "ec2cce5093090a6265ce60b07004347b"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.532552 26837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "197e7702e73a4993841d536cf7861d4a"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "b8a019711264d26eab503e6d78d73b48"
server_key_iv: "ec2cce5093090a6265ce60b07004347b"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.537710 26837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:36:10.541121 26852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.542455 26837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:10.542603 26837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "197e7702e73a4993841d536cf7861d4a"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "b8a019711264d26eab503e6d78d73b48"
server_key_iv: "ec2cce5093090a6265ce60b07004347b"
server_key_version: "encryptionkey@0"
I20260929 16:36:10.542785 26837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:10.561173 26837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:10.565409 26837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:10.565670 26837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:10.566638 26837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:10.567977 26837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:10.568050 26837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.568182 26837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:10.568233 26837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:10.597618 26837 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:35005
I20260929 16:36:10.597657 26964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:35005 every 8 connection(s)
I20260929 16:36:10.598894 26837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:10.602774 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26837
I20260929 16:36:10.602931 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:36:10.603396 18228 external_mini_cluster.cc:1428] Setting key 928a335b384ef844817a144752fd1162
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699770, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:10.617892 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.601355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:42355 (local address 127.17.205.62:42549)
0929 16:36:10.601507 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:10.601512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:10.602672 (+ 1160us) server_negotiation.cc:407] Connection header received
0929 16:36:10.604627 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:10.604632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:10.604718 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:10.604797 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:10.606456 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.607237 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.608724 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.609096 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.609208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.612512 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:10.612517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:10.612520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:10.612552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:10.614929 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.615646 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.615652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.615655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.615765 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.616157 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.616160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.616162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.616435 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:10.616559 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:10.617395 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 16:36:10.617566 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:36:10.619053 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.601834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:42355)
0929 16:36:10.602517 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:10.602556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:10.604368 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:10.605060 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:10.605073 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:10.605609 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:10.606308 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:10.606331 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.607372 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.607378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:10.608596 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:10.608607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.609332 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.609340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:10.609570 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.610168 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:10.610194 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:10.612317 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:10.615054 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:10.615063 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:10.615082 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:10.615541 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:10.615862 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:10.615867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:10.615870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:10.616048 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:10.616549 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:10.616558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:10.616971 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 16:36:10.617251 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:36:10.617787 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":247,"threads_started":1}
I20260929 16:36:10.620452 26965 heartbeater.cc:347] Connected to a master server at 127.17.205.62:42549
I20260929 16:36:10.620774 26965 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:10.621592 26965 heartbeater.cc:511] Master 127.17.205.62:42549 requested a full tablet report, sending...
I20260929 16:36:10.623018 26509 ts_manager.cc:194] Registered new tserver with Master: 197e7702e73a4993841d536cf7861d4a (127.17.205.3:35005)
I20260929 16:36:10.623850 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:42355
I20260929 16:36:10.631155 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.1@KRBTEST.COM" created.
I20260929 16:36:10.673657 26698 heartbeater.cc:503] Master 127.17.205.62:42549 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
Entry for principal oryx/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab.
WARNING: no policy specified for oryx/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
Entry for principal oryx/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab.
WARNING: no policy specified for oryx/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
Entry for principal oryx/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab.
I20260929 16:36:10.753563 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26570
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:36:10.768191 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.754609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.17.205.1:45201)
0929 16:36:10.755296 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:10.755303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:10.755330 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:36:10.755394 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:10.755401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:10.755611 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:10.755760 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:10.756917 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.757882 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.759047 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.759630 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.759734 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.763259 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:10.763275 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:10.763289 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:10.763322 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:10.765928 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.766470 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.766476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.766478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.766594 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.766869 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.766871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.766873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.767185 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:10.767431 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:10.767643 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:36:10.767841 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260929 16:36:10.769409 26693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:10.850196 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26570
I20260929 16:36:10.863829 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26704
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:36:10.878798 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.864713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.17.205.2:38469)
0929 16:36:10.865186 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:10.865194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:10.865228 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:36:10.865284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:10.865290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:10.865500 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:10.865646 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:10.866828 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.867866 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.869057 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.869585 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.869694 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.873534 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:10.873545 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:10.873560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:10.873593 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:10.876245 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.876863 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.876870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.876872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.876986 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.877319 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.877322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.877324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.877653 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:10.877929 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:10.878231 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 16:36:10.878473 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260929 16:36:10.880211 26827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:10.960632 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26704
I20260929 16:36:10.973492 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26837
Sep 29 16:36:10 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:36:10.988855 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:10.974394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.17.205.3:35005)
0929 16:36:10.974936 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:10.974942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:10.974972 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:36:10.975028 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:10.975034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:10.975295 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:10.975423 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:10.976650 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.977964 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.979140 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:10.979759 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:10.979855 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:10.983816 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:10.983828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:10.983844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:10.983878 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:10.986459 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.987053 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.987059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.987061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.987171 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:10.987554 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:10.987557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:10.987559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:10.987882 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:10.988109 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:10.988325 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:36:10.988514 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260929 16:36:10.990053 26960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:11.070971 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26837
I20260929 16:36:11.083791 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26477
I20260929 16:36:11.084879 26539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:11.183986 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26477
I20260929 16:36:11.201159 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42549
--webserver_interface=127.17.205.62
--webserver_port=32787
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:11.524271 26982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:11.524569 26982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:11.524662 26982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:11.531837 26982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:11.531949 26982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:11.532001 26982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:11.532054 26982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:11.532096 26982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:11.541945 26982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:42549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=32787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:11.543282 26982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:11.544582 26982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:11.555769 26987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:11.555703 26988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:11.555676 26990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:11.556473 26982 server_base.cc:1034] running on GCE node
I20260929 16:36:11.557222 26982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:11.558677 26982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:11.559880 26982 hybrid_clock.cc:648] HybridClock initialized: now 1790699771559820 us; error 48 us; skew 500 ppm
I20260929 16:36:11.560237 26982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:11 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699771, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:11.563184 26982 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:36:11.564693 26982 webserver.cc:460] Webserver started at http://127.17.205.62:32787/ using document root <none> and password file <none>
I20260929 16:36:11.565384 26982 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:11.565508 26982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:11.570900 26982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:36:11.573670 26997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:11.575053 26982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:36:11.575245 26982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "cf03a4115b0344b88146b6f452ca3d53"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "acc5019a5ea65341fc9bc58c078d42e0"
server_key_iv: "8112d54469912379df32bb424dbf9b4f"
server_key_version: "encryptionkey@0"
I20260929 16:36:11.576187 26982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:11.594803 26982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:11.598987 26982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:11.599277 26982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:11.616564 26982 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:42549
I20260929 16:36:11.616582 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:42549 every 8 connection(s)
I20260929 16:36:11.618101 26982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:11.621367 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 26982
I20260929 16:36:11.622287 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:45201
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:42549
--webserver_port=34729
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:36:11.626382 27049 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:36:11.630585 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Bootstrap starting.
I20260929 16:36:11.649150 27049 log.cc:824] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:11.657815 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:36:11.658293 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Bootstrap complete.
I20260929 16:36:11.667829 27049 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:11.668992 27049 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf03a4115b0344b88146b6f452ca3d53, State: Initialized, Role: FOLLOWER
I20260929 16:36:11.669720 27049 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:11.669993 27049 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:11.670120 27049 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:11.670301 27049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:36:11.674727 27049 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:11.675251 27049 leader_election.cc:302] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf03a4115b0344b88146b6f452ca3d53; no voters:
I20260929 16:36:11.675819 27049 leader_election.cc:288] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:36:11.675963 27053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:36:11.676982 27053 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [term 2 LEADER]: Becoming Leader. State: Replica: cf03a4115b0344b88146b6f452ca3d53, State: Running, Role: LEADER
I20260929 16:36:11.677603 27053 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } }
I20260929 16:36:11.677917 27049 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:11.685252 27055 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf03a4115b0344b88146b6f452ca3d53. Latest consensus state: current_term: 2 leader_uuid: "cf03a4115b0344b88146b6f452ca3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } } }
I20260929 16:36:11.685653 27055 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:11.686137 27054 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf03a4115b0344b88146b6f452ca3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf03a4115b0344b88146b6f452ca3d53" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42549 } } }
I20260929 16:36:11.686417 27054 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:11.688396 27061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:11.694898 27061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:11.696378 27061 catalog_manager.cc:1259] Loaded cluster ID: cf438691d63f47a088200a9975fc97cd
I20260929 16:36:11.696539 27061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:11.700445 27061 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:11.702162 27061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf03a4115b0344b88146b6f452ca3d53: Loaded TSK: 0
I20260929 16:36:11.703826 27061 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:36:11.967608 27051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:11.967890 27051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:11.968009 27051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:11.975196 27051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:11.975353 27051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:11.975503 27051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:11.985536 27051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.1.keytab
--principal=oryx/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:45201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=34729
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:11.986856 27051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:11.988225 27051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:12.002940 27076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.002976 27077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.004112 27079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:12.004908 27051 server_base.cc:1034] running on GCE node
I20260929 16:36:12.005672 27051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:12.006901 27051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:12.008134 27051 hybrid_clock.cc:648] HybridClock initialized: now 1790699772008079 us; error 46 us; skew 500 ppm
I20260929 16:36:12.008529 27051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699772, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:12.012569 27051 init.cc:377] Logged in from keytab as oryx/127.17.205.1@KRBTEST.COM (short username oryx)
I20260929 16:36:12.014424 27051 webserver.cc:460] Webserver started at http://127.17.205.1:34729/ using document root <none> and password file <none>
I20260929 16:36:12.015374 27051 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:12.015516 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:12.022544 27051 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 16:36:12.025928 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.027642 27051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:36:12.027839 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "db5dfccc2d4547dead919dfa7fb86580"
format_stamp: "Formatted at 2026-09-29 16:36:09 on dist-test-slave-3h01"
server_key: "dbf1153a9aea6d5d541cd4f3706be8e0"
server_key_iv: "78cddb79a5c16e9879ca20e5e1b9af37"
server_key_version: "encryptionkey@0"
I20260929 16:36:12.029176 27051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:12.061682 27051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:12.066160 27051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:12.066426 27051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:12.067461 27051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:12.068822 27051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:12.068915 27051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.069020 27051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:12.069090 27051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.098177 27051 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:45201
I20260929 16:36:12.098243 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:45201 every 8 connection(s)
I20260929 16:36:12.099736 27051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:12.105679 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27051
I20260929 16:36:12.106251 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:38469
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:42549
--webserver_port=44859
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:36:12.423391 27203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:12.423694 27203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:12.423832 27203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:12.430823 27203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:12.430919 27203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:12.431073 27203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:12.440614 27203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.2.keytab
--principal=oryx/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:38469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=44859
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:12.441872 27203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:12.443181 27203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:12.458756 27210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.458943 27209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.458787 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:12.464581 27203 server_base.cc:1034] running on GCE node
I20260929 16:36:12.465509 27203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:12.466938 27203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:12.468195 27203 hybrid_clock.cc:648] HybridClock initialized: now 1790699772468083 us; error 95 us; skew 500 ppm
I20260929 16:36:12.468657 27203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699772, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:12.473117 27203 init.cc:377] Logged in from keytab as oryx/127.17.205.2@KRBTEST.COM (short username oryx)
I20260929 16:36:12.475118 27203 webserver.cc:460] Webserver started at http://127.17.205.2:44859/ using document root <none> and password file <none>
I20260929 16:36:12.476241 27203 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:12.476401 27203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:12.484520 27203 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:36:12.488140 27219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.489944 27203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:36:12.490134 27203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "e556a951d2bd4ff590aa8a148b7c631d"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "70306b65f4a1c0c66c750ab4191aba95"
server_key_iv: "5850b66c5e878f02ceee15ec8abd5eb3"
server_key_version: "encryptionkey@0"
I20260929 16:36:12.491627 27203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:12.515755 27203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:12.521726 27203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:12.522068 27203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:12.523465 27203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:12.525430 27203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:12.525554 27203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.525668 27203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:12.525753 27203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.561877 27203 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38469
I20260929 16:36:12.561887 27331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38469 every 8 connection(s)
I20260929 16:36:12.564055 27203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:12.566249 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27203
I20260929 16:36:12.566808 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:35005
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:42549
--webserver_port=37153
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:36:12.899850 27336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:12.900144 27336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:12.900285 27336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:12.907728 27336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:12.907856 27336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:12.908039 27336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:12.917966 27336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/krb5kdc/oryx_127.17.205.3.keytab
--principal=oryx/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:35005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=37153
--tserver_master_addrs=127.17.205.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:12.919402 27336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:12.920758 27336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:12.935837 27342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.935874 27343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:12.949062 27345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:12.949750 27336 server_base.cc:1034] running on GCE node
I20260929 16:36:12.950737 27336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:12.952286 27336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:12.953649 27336 hybrid_clock.cc:648] HybridClock initialized: now 1790699772953559 us; error 71 us; skew 500 ppm
I20260929 16:36:12.954126 27336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:12 dist-test-slave-3h01 krb5kdc[26464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699772, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:12.958757 27336 init.cc:377] Logged in from keytab as oryx/127.17.205.3@KRBTEST.COM (short username oryx)
I20260929 16:36:12.961177 27336 webserver.cc:460] Webserver started at http://127.17.205.3:37153/ using document root <none> and password file <none>
I20260929 16:36:12.962324 27336 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:12.962502 27336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:12.971407 27336 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:36:12.975785 27352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:12.978034 27336 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:36:12.978264 27336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "197e7702e73a4993841d536cf7861d4a"
format_stamp: "Formatted at 2026-09-29 16:36:10 on dist-test-slave-3h01"
server_key: "b8a019711264d26eab503e6d78d73b48"
server_key_iv: "ec2cce5093090a6265ce60b07004347b"
server_key_version: "encryptionkey@0"
I20260929 16:36:12.979883 27336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:13.024061 27336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:13.030190 27336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:13.030545 27336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:13.032051 27336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:13.034127 27336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:13.034258 27336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:13.034420 27336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:13.034504 27336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:13.085419 27336 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:35005
I20260929 16:36:13.085562 27464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:35005 every 8 connection(s)
I20260929 16:36:13.087815 27336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:13.096556 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27336
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:13.133061 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.097787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.17.205.62:42549)
0929 16:36:13.106646 (+ 8859us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.106654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.110378 (+ 3724us) server_negotiation.cc:407] Connection header received
0929 16:36:13.110943 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.110952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.111051 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.111140 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.113855 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.115278 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.117452 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.118178 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.119812 (+ 1634us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.122297 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:13.122310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:13.122339 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:13.122403 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:13.126607 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:13.127657 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:13.127668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:13.127680 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:13.127961 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:13.128393 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:13.128398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:13.128401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:13.129065 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:13.129714 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:13.131282 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0929 16:36:13.131840 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":8698,"thread_start_us":13443,"threads_started":1}
W20260929 16:36:13.292502 27202 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.180s user 0.000s sys 0.011s
W20260929 16:36:13.294343 27201 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:12.102744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:59641 (local address 127.17.205.62:42549)
0929 16:36:12.103531 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:12.103547 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:12.103805 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:36:12.105893 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:12.105907 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:12.106244 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:12.106631 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:12.108139 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.109368 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.110880 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.111853 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.112042 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.293960 (+1181918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:59641: BlockingRecv error: recv got EOF from 127.17.205.1:59641 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":244,"threads_started":1}
I20260929 16:36:13.295311 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:12.102836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:59641)
0929 16:36:12.103644 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:12.103685 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:12.105322 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:12.106769 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:12.106780 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:12.107408 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:12.107959 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:12.107977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.109514 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.109518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:12.110746 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:12.110758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.112067 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.112073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:12.112290 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:12.112907 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:12.112929 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.293571 (+1180642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":372,"threads_started":1}
W20260929 16:36:13.296762 27199 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:42549 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.205.62:42549: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:36:13.311017 27201 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:13.298025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:33241 (local address 127.17.205.62:42549)
0929 16:36:13.298284 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.298291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.298312 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:36:13.298375 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.298381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.298466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.298580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.299628 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.300960 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.303184 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.303904 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.304047 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.310572 (+ 6525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:33241: BlockingRecv error: recv got EOF from 127.17.205.1:33241 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260929 16:36:13.312001 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.297582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:33241)
0929 16:36:13.297821 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:13.297847 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:13.297998 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:13.298628 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:13.298634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:13.299144 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:13.299435 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.299447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.301129 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.301136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.303023 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.303043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.304094 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.304101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.304308 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.305106 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:13.305132 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.311668 (+ 6536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:13.326406 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.312946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:47473)
0929 16:36:13.313654 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:13.313678 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:13.313879 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:13.314295 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:13.314300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:13.314956 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:13.315330 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.315345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.316857 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.316863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.318526 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.318544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.319647 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.319654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.319858 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.320697 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:13.320727 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.326000 (+ 5273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":594,"threads_started":1}
W20260929 16:36:13.326587 27201 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:13.313388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:47473 (local address 127.17.205.62:42549)
0929 16:36:13.313568 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.313574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.313762 (+ 188us) server_negotiation.cc:407] Connection header received
0929 16:36:13.314027 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.314033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.314135 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.314258 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.315491 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.316710 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.318688 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.319460 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.319596 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.326265 (+ 6669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:47473: BlockingRecv error: recv got EOF from 127.17.205.1:47473 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 16:36:13.327169 27199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:36:13.716027 27335 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.121s user 0.005s sys 0.012s
W20260929 16:36:13.717403 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:12.567027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:45683 (local address 127.17.205.62:42549)
0929 16:36:12.585608 (+ 18581us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:12.585614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:12.585635 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:36:12.585696 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:12.585701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:12.585777 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:12.585894 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:12.588653 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.589803 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.593123 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.593747 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.593883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.717037 (+1123154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:45683: BlockingRecv error: recv got EOF from 127.17.205.2:45683 (error 108)
Metrics: {"server-negotiator.queue_time_us":18451,"thread_start_us":18962,"threads_started":1}
I20260929 16:36:13.717630 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:12.567844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:45683)
0929 16:36:12.577794 (+ 9950us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:12.577857 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:12.580247 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:12.586362 (+ 6115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:12.586379 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:12.587374 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:12.588397 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:12.588428 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.590525 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.590531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:12.592922 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:12.592940 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:12.594006 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:12.594013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:12.594363 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:12.595247 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:12.595282 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.716752 (+1121470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9606,"thread_start_us":9377,"threads_started":1}
W20260929 16:36:13.718485 27332 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:42549 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.205.62:42549: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:13.728120 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.719143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:57823)
0929 16:36:13.719398 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:13.719417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:13.719531 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:13.719815 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:13.719819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:13.720177 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:13.720335 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.720344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.721360 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.721366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.722325 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.722337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.723046 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.723052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.723244 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.723949 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:13.723965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.727834 (+ 3869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 16:36:13.728385 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:13.719159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57823 (local address 127.17.205.62:42549)
0929 16:36:13.719330 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.719337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.719477 (+ 140us) server_negotiation.cc:407] Connection header received
0929 16:36:13.719639 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.719643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.719699 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.719775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.720474 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.721217 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.722464 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.722880 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.722989 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.728086 (+ 5097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:57823: BlockingRecv error: recv got EOF from 127.17.205.2:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:13 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:13.738591 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.729086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:33545)
0929 16:36:13.729394 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:13.729411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:13.729511 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:13.730009 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:13.730015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:13.730432 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:13.730622 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.730631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.732173 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.732177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.733106 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.733117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.734099 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.734105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.734245 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.734913 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:13.734936 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:13.738390 (+ 3454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204}
W20260929 16:36:13.738986 27332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:36:13.739050 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:13.729241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33545 (local address 127.17.205.62:42549)
0929 16:36:13.729410 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.729417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.729461 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:36:13.729627 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.729633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.729739 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.729847 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.730759 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.732009 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.733231 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.733927 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.734132 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.738742 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:33545: BlockingRecv error: recv got EOF from 127.17.205.2:33545 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 16:36:14.143846 27468 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.014s user 0.003s sys 0.013s
W20260929 16:36:14.144857 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:13.091111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:43597 (local address 127.17.205.62:42549)
0929 16:36:13.099617 (+ 8506us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:13.099625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:13.112108 (+ 12483us) server_negotiation.cc:407] Connection header received
0929 16:36:13.115092 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:13.115100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:13.115201 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:13.115307 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:13.118264 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.119678 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.127180 (+ 7502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.127922 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.128023 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.144660 (+1016637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:43597: BlockingRecv error: recv got EOF from 127.17.205.3:43597 (error 108)
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":8359,"thread_start_us":9615,"threads_started":1}
I20260929 16:36:14.145390 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:13.091989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:43597)
0929 16:36:13.111951 (+ 19962us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:13.112002 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:13.114675 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:13.116138 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:13.116153 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:13.117043 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:13.118013 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.118040 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.120097 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.120105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.126999 (+ 6894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:13.127020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:13.128453 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:13.128461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:13.128842 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:13.129618 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:13.129661 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:14.144504 (+1014843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19622,"thread_start_us":19389,"threads_started":1}
W20260929 16:36:14.146194 27465 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:42549 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.205.62:42549: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:14.159420 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:14.146820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:50421)
0929 16:36:14.147006 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:14.147026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:14.147150 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:14.147489 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:14.147494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:14.147854 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:14.148007 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.148015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.148999 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.149005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.149951 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.149962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.150635 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.150642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.150800 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.156018 (+ 5218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:14.156037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:14.159154 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:36:14.159572 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:14.146887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:50421 (local address 127.17.205.62:42549)
0929 16:36:14.147032 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:14.147038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:14.147077 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:36:14.147307 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:14.147311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:14.147372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:14.147450 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:14.148136 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.148860 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.150080 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.150479 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.150582 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.159390 (+ 8808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:50421: BlockingRecv error: recv got EOF from 127.17.205.3:50421 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:14.167747 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:14.160038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:45911)
0929 16:36:14.160163 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:14.160178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:14.160280 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:14.160556 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:14.160559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:14.160820 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:14.160981 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.160989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.161946 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.161950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.162870 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.162881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.163567 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.163572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.163693 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.164155 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:14.164170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:14.167472 (+ 3302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 16:36:14.167908 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:14.160113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45911 (local address 127.17.205.62:42549)
0929 16:36:14.160333 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:14.160337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:14.160352 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:36:14.160394 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:14.160397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:14.160452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:14.160526 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:14.161086 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.161836 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.162985 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.163449 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.163537 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.167669 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:45911: BlockingRecv error: recv got EOF from 127.17.205.3:45911 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260929 16:36:14.168049 27465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:14.336602 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:14.327863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:37873)
0929 16:36:14.328388 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:14.328405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:14.328555 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:14.328905 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:14.328910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:14.329315 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:14.329473 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.329482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.330464 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.330469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.331441 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.331453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.332128 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.332134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.332302 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.332877 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:14.332899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:14.336301 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
W20260929 16:36:14.336705 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:14.327978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:37873 (local address 127.17.205.62:42549)
0929 16:36:14.328170 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:14.328178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:14.328497 (+ 319us) server_negotiation.cc:407] Connection header received
0929 16:36:14.328675 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:14.328679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:14.328767 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:14.328854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:14.329629 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.330317 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.331582 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.331982 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.332092 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.336496 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:37873: BlockingRecv error: recv got EOF from 127.17.205.1:37873 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:14 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:14.747886 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:14.739592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:37261)
0929 16:36:14.740090 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:14.740105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:14.740218 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:14.740482 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:14.740485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:14.740902 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:14.741044 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.741053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.742000 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.742003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.742952 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:14.742964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.743651 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.743656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:14.743784 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.744335 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:14.744352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:14.747624 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":280,"threads_started":1}
W20260929 16:36:14.748066 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:14.739655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:37261 (local address 127.17.205.62:42549)
0929 16:36:14.739817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:14.739821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:14.740167 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:36:14.740309 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:14.740313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:14.740372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:14.740453 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:14.741151 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.741887 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.743074 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:14.743509 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:14.743599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:14.747839 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:37261: BlockingRecv error: recv got EOF from 127.17.205.2:37261 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:15.177208 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:15.168633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:32863)
0929 16:36:15.169150 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:15.169166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:15.169288 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:15.169632 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:15.169637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:15.170125 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:15.170283 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.170292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.171281 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.171286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.172217 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.172228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.172957 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.172963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.173113 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.173644 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:15.173664 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:15.176978 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260929 16:36:15.177399 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:15.168713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:32863 (local address 127.17.205.62:42549)
0929 16:36:15.168890 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:15.168894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:15.169238 (+ 344us) server_negotiation.cc:407] Connection header received
0929 16:36:15.169415 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:15.169420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:15.169497 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:15.169576 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:15.170416 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.171132 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.172352 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.172793 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.172898 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.177183 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:32863: BlockingRecv error: recv got EOF from 127.17.205.3:32863 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:15.346287 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:15.337606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:40771)
0929 16:36:15.338062 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:15.338079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:15.338230 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:15.338576 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:15.338580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:15.338945 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:15.339160 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.339172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.340426 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.340430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.341374 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.341386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.342030 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.342034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.342154 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.342769 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:15.342786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:15.345967 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":246,"threads_started":1}
W20260929 16:36:15.346474 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:15.337693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:40771 (local address 127.17.205.62:42549)
0929 16:36:15.337881 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:15.337885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:15.338168 (+ 283us) server_negotiation.cc:407] Connection header received
0929 16:36:15.338366 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:15.338370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:15.338433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:15.338516 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:15.339322 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.340281 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.341506 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.341903 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.341991 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.346240 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:40771: BlockingRecv error: recv got EOF from 127.17.205.1:40771 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:15 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:15.757311 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:15.748924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:33847)
0929 16:36:15.749393 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:15.749409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:15.749523 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:15.749862 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:15.749866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:15.750276 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:15.750453 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.750462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.751429 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.751434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.752342 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:15.752353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.753071 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.753075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:15.753194 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.753758 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:15.753774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:15.757052 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
W20260929 16:36:15.757452 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:15.748961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33847 (local address 127.17.205.62:42549)
0929 16:36:15.749153 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:15.749161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:15.749511 (+ 350us) server_negotiation.cc:407] Connection header received
0929 16:36:15.749619 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:15.749622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:15.749689 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:15.749784 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:15.750593 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.751316 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.752517 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:15.752949 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:15.753052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:15.757246 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:33847: BlockingRecv error: recv got EOF from 127.17.205.2:33847 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:16.187106 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:16.178071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:34363)
0929 16:36:16.178618 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:16.178634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:16.178769 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:16.179101 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:16.179106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:16.179610 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:16.179795 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.179805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.180856 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.180859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.181780 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.181792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.182572 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.182578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.182750 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.183400 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:16.183423 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:16.186775 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":292,"threads_started":1}
W20260929 16:36:16.187250 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:16.178257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:34363 (local address 127.17.205.62:42549)
0929 16:36:16.178466 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:16.178473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:16.178723 (+ 250us) server_negotiation.cc:407] Connection header received
0929 16:36:16.178884 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:16.178889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:16.178968 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:16.179049 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:16.179932 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.180709 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.181913 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.182411 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.182532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.186993 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:34363: BlockingRecv error: recv got EOF from 127.17.205.3:34363 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:16.356072 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:16.347264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:41243)
0929 16:36:16.348103 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:16.348125 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:16.348280 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:16.348607 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:16.348612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:16.348988 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:16.349158 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.349167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.350151 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.350155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.351062 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.351073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.351771 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.351777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.351940 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.352463 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:16.352479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:16.355845 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":574,"threads_started":1}
W20260929 16:36:16.356207 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:16.347447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:41243 (local address 127.17.205.62:42549)
0929 16:36:16.347713 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:16.347719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:16.348244 (+ 525us) server_negotiation.cc:407] Connection header received
0929 16:36:16.348408 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:16.348414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:16.348476 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:16.348562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:16.349297 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.350018 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.351195 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.351639 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.351744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.356016 (+ 4272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:41243: BlockingRecv error: recv got EOF from 127.17.205.1:41243 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:16 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:16.766430 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:16.758330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:39813)
0929 16:36:16.758808 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:16.758824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:16.758933 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:16.759183 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:16.759186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:16.759587 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:16.759747 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.759755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.760743 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.760747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.761660 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:16.761673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.762305 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.762309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:16.762424 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.762982 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:16.762999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:16.766198 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
W20260929 16:36:16.766609 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:16.758431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:39813 (local address 127.17.205.62:42549)
0929 16:36:16.758585 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:16.758590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:16.758885 (+ 295us) server_negotiation.cc:407] Connection header received
0929 16:36:16.759015 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:16.759018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:16.759077 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:16.759154 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:16.759859 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.760608 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.761775 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:16.762178 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:16.762283 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:16.766387 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:39813: BlockingRecv error: recv got EOF from 127.17.205.2:39813 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:17.196871 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:17.188058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:51063)
0929 16:36:17.188567 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:17.188584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:17.188730 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:17.189091 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:17.189097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:17.189563 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:17.189779 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.189794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.190764 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.190771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.192028 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.192042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.192667 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.192674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.192805 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.193401 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:17.193420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:17.196610 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
W20260929 16:36:17.197052 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:17.188196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:51063 (local address 127.17.205.62:42549)
0929 16:36:17.188418 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:17.188425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:17.188684 (+ 259us) server_negotiation.cc:407] Connection header received
0929 16:36:17.188859 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:17.188865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:17.188951 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:17.189045 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:17.189919 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.190617 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.192145 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.192540 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.192630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.196859 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:51063: BlockingRecv error: recv got EOF from 127.17.205.3:51063 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:17.365669 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:17.357115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:56791)
0929 16:36:17.357621 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:17.357638 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:17.357779 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:17.358089 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:17.358093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:17.358554 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:17.358761 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.358773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.359961 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.359965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.360953 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.360965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.361640 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.361646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.361824 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.362324 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:17.362341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:17.365409 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
W20260929 16:36:17.365815 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:17.357233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:56791 (local address 127.17.205.62:42549)
0929 16:36:17.357466 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:17.357472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:17.357721 (+ 249us) server_negotiation.cc:407] Connection header received
0929 16:36:17.357896 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:17.357901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:17.357961 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:17.358051 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:17.358902 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.359831 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.361082 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.361475 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.361607 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.365604 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:56791: BlockingRecv error: recv got EOF from 127.17.205.1:56791 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:17 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:17.775804 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:17.767514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:38827)
0929 16:36:17.768000 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:17.768016 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:17.768144 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:17.768495 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:17.768499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:17.768944 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:17.769124 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.769137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.770246 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.770250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.771202 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:17.771252 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.771883 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.771888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:17.772045 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.772599 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:17.772616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:17.775558 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
W20260929 16:36:17.775938 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:17.767606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:38827 (local address 127.17.205.62:42549)
0929 16:36:17.767768 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:17.767773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:17.768101 (+ 328us) server_negotiation.cc:407] Connection header received
0929 16:36:17.768259 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:17.768264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:17.768333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:17.768421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:17.769268 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.770121 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.771347 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:17.771737 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:17.771831 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:17.775733 (+ 3902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:38827: BlockingRecv error: recv got EOF from 127.17.205.2:38827 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:18.206763 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:18.197899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:47405)
0929 16:36:18.198454 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:18.198470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:18.198606 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:18.198978 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:18.198983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:18.199401 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:18.199605 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.199617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.200659 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.200664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.201946 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.201959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.202632 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.202637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.202809 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.203382 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:18.203400 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:18.206515 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
W20260929 16:36:18.206936 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:18.197984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:47405 (local address 127.17.205.62:42549)
0929 16:36:18.198238 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:18.198245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:18.198560 (+ 315us) server_negotiation.cc:407] Connection header received
0929 16:36:18.198746 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:18.198750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:18.198827 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:18.198915 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:18.199772 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.200522 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.202078 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.202488 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.202579 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.206706 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:47405: BlockingRecv error: recv got EOF from 127.17.205.3:47405 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:18.374951 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:18.366501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:54321)
0929 16:36:18.366986 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:18.367003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:18.367122 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:18.367453 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:18.367456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:18.367839 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:18.367996 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.368008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.369207 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.369212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.370249 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.370262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.370922 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.370928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.371087 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.371636 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:18.371653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:18.374622 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
W20260929 16:36:18.375074 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:18.366617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:54321 (local address 127.17.205.62:42549)
0929 16:36:18.366771 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:18.366775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:18.367092 (+ 317us) server_negotiation.cc:407] Connection header received
0929 16:36:18.367262 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:18.367267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:18.367333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:18.367418 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:18.368144 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.369067 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.370392 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.370773 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.370876 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.374839 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:54321: BlockingRecv error: recv got EOF from 127.17.205.1:54321 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:18 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:18.785655 27486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:18.776687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:39955)
0929 16:36:18.777181 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:18.777197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:18.777370 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:18.777773 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:18.777778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:18.778206 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:18.778388 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.778398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.779618 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.779622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.780770 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:18.780785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.781454 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.781459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:18.781581 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.782181 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:18.782200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:18.785436 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
W20260929 16:36:18.785882 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:18.776931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:39955 (local address 127.17.205.62:42549)
0929 16:36:18.777140 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:18.777146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:18.777265 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:36:18.777565 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:18.777571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:18.777642 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:18.777726 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:18.778518 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.779499 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.780886 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:18.781336 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:18.781436 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:18.785631 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:39955: BlockingRecv error: recv got EOF from 127.17.205.2:39955 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:19.220888 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:19.207670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:37297)
0929 16:36:19.208140 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:19.208157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:19.208283 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:19.208705 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:19.208711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:19.209120 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:19.209262 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.209270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.210235 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.210239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.211161 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.211172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.211846 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.211852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.212006 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.217412 (+ 5406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:19.217431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:19.220644 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
W20260929 16:36:19.221062 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:19.207782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:37297 (local address 127.17.205.62:42549)
0929 16:36:19.207966 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:19.207973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:19.208231 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:36:19.208483 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:19.208489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:19.208572 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:19.208661 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:19.209397 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.210105 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.211306 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.211709 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.211817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.220837 (+ 9020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:37297: BlockingRecv error: recv got EOF from 127.17.205.3:37297 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:19.385313 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:19.375988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:45637)
0929 16:36:19.376483 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:19.376499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:19.376632 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:19.376951 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:19.376955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:19.377380 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:19.377532 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.377541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.378610 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.378616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.379588 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.379599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.380341 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.380347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.380512 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.381308 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:19.381333 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:19.385057 (+ 3724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
W20260929 16:36:19.385500 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:19.376071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:45637 (local address 127.17.205.62:42549)
0929 16:36:19.376268 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:19.376273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:19.376584 (+ 311us) server_negotiation.cc:407] Connection header received
0929 16:36:19.376744 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:19.376748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:19.376809 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:19.376903 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:19.377725 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.378464 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.379714 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.380188 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.380342 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.385266 (+ 4924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:45637: BlockingRecv error: recv got EOF from 127.17.205.1:45637 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:19 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:19.794986 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:19.786613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:33921)
0929 16:36:19.787058 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:19.787076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:19.787238 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:19.787531 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:19.787534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:19.787909 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:19.788070 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.788079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.789190 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.789196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.790209 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:19.790222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.790911 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.790917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:19.791078 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.791663 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:19.791680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:19.794751 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
W20260929 16:36:19.795145 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:19.786655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33921 (local address 127.17.205.62:42549)
0929 16:36:19.786829 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:19.786834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:19.787144 (+ 310us) server_negotiation.cc:407] Connection header received
0929 16:36:19.787357 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:19.787362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:19.787425 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:19.787499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:19.788212 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.789039 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.790344 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:19.790753 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:19.790873 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:19.794956 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:33921: BlockingRecv error: recv got EOF from 127.17.205.2:33921 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:20.229687 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:20.221734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:46345)
0929 16:36:20.222211 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:20.222229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:20.222348 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:20.222641 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:20.222644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:20.223031 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:20.223189 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.223198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.224176 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.224179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.225127 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.225139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.225762 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.225767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.225884 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.226401 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:20.226417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:20.229452 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":266,"threads_started":1}
W20260929 16:36:20.229864 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:20.221811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:46345 (local address 127.17.205.62:42549)
0929 16:36:20.221962 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:20.221967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:20.222292 (+ 325us) server_negotiation.cc:407] Connection header received
0929 16:36:20.222435 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:20.222439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:20.222511 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:20.222596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:20.223332 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.224065 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.225245 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.225640 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.225731 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.229647 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:46345: BlockingRecv error: recv got EOF from 127.17.205.3:46345 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:20.395473 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:20.386207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:51459)
0929 16:36:20.386862 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:20.386882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:20.387022 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:20.387415 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:20.387420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:20.387814 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:20.387985 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.387994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.388936 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.388939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.389854 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.389864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.390477 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.390481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.390589 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.391118 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:20.391134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:20.395100 (+ 3966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
W20260929 16:36:20.395571 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:20.386321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:51459 (local address 127.17.205.62:42549)
0929 16:36:20.386531 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:20.386539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:20.386962 (+ 423us) server_negotiation.cc:407] Connection header received
0929 16:36:20.387140 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:20.387145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:20.387244 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:20.387370 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:20.388108 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.388840 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.389959 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.390366 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.390453 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.395361 (+ 4908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:51459: BlockingRecv error: recv got EOF from 127.17.205.1:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:20 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:20.804523 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:20.795853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:32989)
0929 16:36:20.796413 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:20.796429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:20.796534 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:20.796794 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:20.796798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:20.797205 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:20.797411 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.797424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.798464 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.798470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.799453 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:20.799465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.800109 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.800113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:20.800295 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.800823 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:20.800838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:20.804201 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":241,"threads_started":1}
W20260929 16:36:20.804636 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:20.795921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:32989 (local address 127.17.205.62:42549)
0929 16:36:20.796088 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:20.796092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:20.796497 (+ 405us) server_negotiation.cc:407] Connection header received
0929 16:36:20.796620 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:20.796624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:20.796683 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:20.796759 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:20.797544 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.798348 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.799576 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:20.799997 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:20.800087 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:20.804390 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:32989: BlockingRecv error: recv got EOF from 127.17.205.2:32989 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:21.238646 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:21.230558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:43135)
0929 16:36:21.231090 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:21.231106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:21.231247 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:21.231534 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:21.231537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:21.231911 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:21.232076 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.232085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.232994 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.232997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.233912 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.233922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.234571 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.234575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.234681 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.235317 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:21.235337 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:21.238401 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":254,"threads_started":1}
W20260929 16:36:21.238849 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:21.230620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:43135 (local address 127.17.205.62:42549)
0929 16:36:21.230808 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:21.230815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:21.231179 (+ 364us) server_negotiation.cc:407] Connection header received
0929 16:36:21.231347 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:21.231352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:21.231426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:21.231506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:21.232178 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.232893 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.234022 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.234456 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.234545 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.238599 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:43135: BlockingRecv error: recv got EOF from 127.17.205.3:43135 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:21.405617 27494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:21.396432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:40235)
0929 16:36:21.397525 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:21.397545 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:21.397698 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:21.398046 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:21.398051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:21.398511 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:21.398704 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.398717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.399719 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.399723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.400653 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.400664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.401344 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.401350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.401492 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.402079 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:21.402096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:21.405278 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":798,"threads_started":1}
W20260929 16:36:21.405725 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:21.396543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:40235 (local address 127.17.205.62:42549)
0929 16:36:21.396696 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:21.396700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:21.397649 (+ 949us) server_negotiation.cc:407] Connection header received
0929 16:36:21.397836 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:21.397841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:21.397918 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:21.398009 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:21.398821 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.399615 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.400758 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.401175 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.401291 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.405495 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:40235: BlockingRecv error: recv got EOF from 127.17.205.1:40235 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:21 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:21.816009 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:21.805724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:56505)
0929 16:36:21.806417 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:21.806439 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:21.806585 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:21.807164 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:21.807170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:21.807756 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:21.807995 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.808010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.809333 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.809340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.810851 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:21.810869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.811756 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.811762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:21.811891 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.812419 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:21.812436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:21.815750 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":395,"threads_started":1}
W20260929 16:36:21.816159 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:21.805786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:56505 (local address 127.17.205.62:42549)
0929 16:36:21.805968 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:21.805975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:21.806539 (+ 564us) server_negotiation.cc:407] Connection header received
0929 16:36:21.806718 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:21.806724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:21.806833 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:21.806928 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:21.808158 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.809177 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.811019 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:21.811556 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:21.811681 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:21.815962 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:56505: BlockingRecv error: recv got EOF from 127.17.205.2:56505 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:22.248241 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:22.239674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:40517)
0929 16:36:22.240192 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:22.240208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:22.240381 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:22.240711 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:22.240716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:22.241135 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:22.241311 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.241320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.242316 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.242320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.243309 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.243322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.244033 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.244038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.244188 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.244740 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:22.244760 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:22.247925 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
W20260929 16:36:22.248378 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:22.239813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:40517 (local address 127.17.205.62:42549)
0929 16:36:22.240015 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:22.240021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:22.240326 (+ 305us) server_negotiation.cc:407] Connection header received
0929 16:36:22.240499 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:22.240504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:22.240578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:22.240661 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:22.241455 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.242182 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.243439 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.243863 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.243971 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.248143 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:40517: BlockingRecv error: recv got EOF from 127.17.205.3:40517 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:22.420161 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:22.406952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:40175)
0929 16:36:22.411622 (+ 4670us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:22.411640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:22.411778 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:22.412063 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:22.412066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:22.412424 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:22.412577 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.412585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.413685 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.413690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.414686 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.414700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.415474 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.415478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.415625 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.416306 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:22.416327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:22.419869 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4559,"thread_start_us":4438,"threads_started":1}
W20260929 16:36:22.420385 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:22.407030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:40175 (local address 127.17.205.62:42549)
0929 16:36:22.407333 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:22.407340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:22.411730 (+ 4390us) server_negotiation.cc:407] Connection header received
0929 16:36:22.411874 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:22.411877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:22.411944 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:22.412020 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:22.412729 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.413513 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.414838 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.415322 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.415416 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.420125 (+ 4709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:40175: BlockingRecv error: recv got EOF from 127.17.205.1:40175 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:22 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:22.825656 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:22.816931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:58869)
0929 16:36:22.817412 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:22.817427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:22.817551 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:22.817876 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:22.817881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:22.818263 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:22.818412 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.818421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.819513 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.819517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.820467 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:22.820481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.821202 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.821208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:22.821379 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.821980 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:22.821999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:22.825409 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
W20260929 16:36:22.825848 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:22.817000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:58869 (local address 127.17.205.62:42549)
0929 16:36:22.817188 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:22.817194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:22.817500 (+ 306us) server_negotiation.cc:407] Connection header received
0929 16:36:22.817669 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:22.817674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:22.817734 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:22.817809 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:22.818561 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.819368 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.820609 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:22.821046 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:22.821159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:22.825616 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:58869: BlockingRecv error: recv got EOF from 127.17.205.2:58869 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:23.257681 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:23.249413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:33629)
0929 16:36:23.249920 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:23.249941 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:23.250083 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:23.250477 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:23.250481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:23.250928 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:23.251117 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.251130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.252072 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.252075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.252999 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.253010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.253630 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.253634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.253748 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.254275 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:23.254291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:23.257452 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":248,"threads_started":1}
W20260929 16:36:23.257843 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:23.249470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:33629 (local address 127.17.205.62:42549)
0929 16:36:23.249675 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:23.249681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:23.250035 (+ 354us) server_negotiation.cc:407] Connection header received
0929 16:36:23.250209 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:23.250214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:23.250311 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:23.250442 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:23.251273 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.251965 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.253102 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.253519 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.253607 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.257650 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:33629: BlockingRecv error: recv got EOF from 127.17.205.3:33629 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:23.429224 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:23.421017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:41567)
0929 16:36:23.421451 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:23.421467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:23.421575 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:23.421828 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:23.421831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:23.422174 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:23.422331 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.422340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.423258 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.423262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.424202 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.424213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.424839 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.424843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.424946 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.425495 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:23.425511 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:23.428912 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":209,"threads_started":1}
W20260929 16:36:23.429365 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:23.421075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:41567 (local address 127.17.205.62:42549)
0929 16:36:23.421259 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:23.421265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:23.421534 (+ 269us) server_negotiation.cc:407] Connection header received
0929 16:36:23.421673 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:23.421676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:23.421736 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:23.421803 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:23.422455 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.423132 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.424302 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.424735 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.424822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.429132 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:41567: BlockingRecv error: recv got EOF from 127.17.205.1:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:23 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:23.834365 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:23.826484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:35181)
0929 16:36:23.826909 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:23.826925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:23.827033 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:23.827314 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:23.827317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:23.827652 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:23.827813 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.827823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.828777 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.828781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.829773 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:23.829785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.830445 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.830449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:23.830554 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.831076 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:23.831092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:23.834128 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
W20260929 16:36:23.834492 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:23.826570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:35181 (local address 127.17.205.62:42549)
0929 16:36:23.826785 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:23.826791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:23.826990 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:36:23.827131 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:23.827135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:23.827196 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:23.827278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:23.827934 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.828664 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.829924 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:23.830318 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:23.830420 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:23.834302 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:35181: BlockingRecv error: recv got EOF from 127.17.205.2:35181 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:24.271148 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:24.258906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:42605)
0929 16:36:24.259453 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:24.259472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:24.259615 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:24.259943 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:24.259948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:24.260335 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:24.260492 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.260500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.261558 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.261564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.262533 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.262544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.263261 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.263268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.263423 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.267613 (+ 4190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:24.267631 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:24.270814 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
W20260929 16:36:24.271361 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:24.259176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:42605 (local address 127.17.205.62:42549)
0929 16:36:24.259445 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:24.259451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:24.259554 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:36:24.259733 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:24.259737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:24.259801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:24.259882 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:24.260636 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.261430 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.262667 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.263092 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.263201 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.271055 (+ 7854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:42605: BlockingRecv error: recv got EOF from 127.17.205.3:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:24.438794 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:24.430245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:42913)
0929 16:36:24.430804 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:24.430822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:24.430967 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:24.431302 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:24.431306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:24.431724 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:24.431902 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.431911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.432916 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.432923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.433818 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.433830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.434559 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.434566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.434726 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.435328 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:24.435346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:24.438568 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":350,"threads_started":1}
W20260929 16:36:24.438941 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:24.430379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:42913 (local address 127.17.205.62:42549)
0929 16:36:24.430544 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:24.430549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:24.430908 (+ 359us) server_negotiation.cc:407] Connection header received
0929 16:36:24.431086 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:24.431090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:24.431154 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:24.431264 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:24.432038 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.432777 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.433956 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.434405 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.434522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.438765 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:42913: BlockingRecv error: recv got EOF from 127.17.205.1:42913 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:24 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:24.844112 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:24.835326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:37865)
0929 16:36:24.835993 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:24.836012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:24.836127 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:24.836480 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:24.836483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:24.836866 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:24.837032 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.837040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.838004 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.838007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.838900 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:24.838910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.839735 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.839742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:24.839892 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.840563 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:24.840587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:24.843823 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":464,"threads_started":1}
W20260929 16:36:24.844241 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:24.835448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:37865 (local address 127.17.205.62:42549)
0929 16:36:24.835647 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:24.835654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:24.836104 (+ 450us) server_negotiation.cc:407] Connection header received
0929 16:36:24.836261 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:24.836266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:24.836358 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:24.836455 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:24.837153 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.837907 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.839035 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:24.839572 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:24.839690 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:24.844016 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:37865: BlockingRecv error: recv got EOF from 127.17.205.2:37865 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:25.280865 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:25.272319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:55121)
0929 16:36:25.272806 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:25.272822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:25.272945 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:25.273323 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:25.273328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:25.273791 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:25.274010 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.274020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.274964 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.274970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.275917 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.275933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.276592 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.276598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.276759 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.277328 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:25.277346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:25.280615 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
W20260929 16:36:25.281037 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:25.272416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:55121 (local address 127.17.205.62:42549)
0929 16:36:25.272583 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:25.272588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:25.272921 (+ 333us) server_negotiation.cc:407] Connection header received
0929 16:36:25.273077 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:25.273081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:25.273168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:25.273257 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:25.274145 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.274840 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.276059 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.276452 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.276558 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.280810 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:55121: BlockingRecv error: recv got EOF from 127.17.205.3:55121 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:25.448616 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:25.439833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:52713)
0929 16:36:25.440379 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:25.440398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:25.440544 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:25.440921 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:25.440925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:25.441411 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:25.441629 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.441643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.442594 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.442598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.443522 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.443532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.444215 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.444220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.444349 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.444888 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:25.444905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:25.448242 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":313,"threads_started":1}
W20260929 16:36:25.448748 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:25.439931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:52713 (local address 127.17.205.62:42549)
0929 16:36:25.440114 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:25.440120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:25.440485 (+ 365us) server_negotiation.cc:407] Connection header received
0929 16:36:25.440664 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:25.440669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:25.440782 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:25.440883 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:25.441766 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.442496 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.443636 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.444067 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.444155 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.448481 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:52713: BlockingRecv error: recv got EOF from 127.17.205.1:52713 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:25 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:25.853488 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:25.845062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:40943)
0929 16:36:25.845608 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:25.845624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:25.845733 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:25.846026 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:25.846029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:25.846409 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:25.846572 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.846581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.847585 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.847589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.848521 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:25.848533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.849173 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.849177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:25.849306 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.849913 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:25.849929 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:25.853244 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
W20260929 16:36:25.853649 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:25.845113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:40943 (local address 127.17.205.62:42549)
0929 16:36:25.845274 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:25.845279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:25.845684 (+ 405us) server_negotiation.cc:407] Connection header received
0929 16:36:25.845836 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:25.845839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:25.845912 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:25.845985 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:25.846678 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.847467 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.848632 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:25.849035 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:25.849126 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:25.853447 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:40943: BlockingRecv error: recv got EOF from 127.17.205.2:40943 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:26.290221 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:26.281739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:59615)
0929 16:36:26.282295 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:26.282310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:26.282444 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:26.282719 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:26.282722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:26.283082 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:26.283285 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.283297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.284242 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.284245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.285153 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.285166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.285803 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.285807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.285937 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.286551 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:26.286567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:26.289980 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":304,"threads_started":1}
W20260929 16:36:26.290381 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:26.281838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:59615 (local address 127.17.205.62:42549)
0929 16:36:26.281996 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:26.282001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:26.282376 (+ 375us) server_negotiation.cc:407] Connection header received
0929 16:36:26.282535 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:26.282538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:26.282611 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:26.282688 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:26.283406 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.284135 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.285275 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.285671 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.285755 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.290159 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:59615: BlockingRecv error: recv got EOF from 127.17.205.3:59615 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:26.458143 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:26.449592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:51881)
0929 16:36:26.450060 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:26.450076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:26.450207 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:26.450508 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:26.450513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:26.450914 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:26.451054 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.451063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.452080 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.452084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.452986 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.452998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.453662 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.453667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.453829 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.454381 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:26.454397 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:26.457907 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
W20260929 16:36:26.458281 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:26.449674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:51881 (local address 127.17.205.62:42549)
0929 16:36:26.449857 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:26.449864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:26.450153 (+ 289us) server_negotiation.cc:407] Connection header received
0929 16:36:26.450315 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:26.450319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:26.450379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:26.450463 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:26.451190 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.451947 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.453117 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.453529 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.453634 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.458089 (+ 4455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:51881: BlockingRecv error: recv got EOF from 127.17.205.1:51881 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:26 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:26.863386 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:26.854602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:57431)
0929 16:36:26.855316 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:26.855333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:26.855465 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:26.855769 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:26.855774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:26.856187 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:26.856325 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.856334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.857320 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.857324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.858218 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:26.858229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.858923 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.858928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:26.859083 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.859718 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:26.859738 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:26.863111 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":473,"threads_started":1}
W20260929 16:36:26.863651 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:26.854779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57431 (local address 127.17.205.62:42549)
0929 16:36:26.855015 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:26.855022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:26.855424 (+ 402us) server_negotiation.cc:407] Connection header received
0929 16:36:26.855577 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:26.855583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:26.855641 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:26.855718 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:26.856468 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.857185 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.858353 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:26.858784 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:26.858890 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:26.863388 (+ 4498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:57431: BlockingRecv error: recv got EOF from 127.17.205.2:57431 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.3@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:27.299350 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:27.291124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.3:33635)
0929 16:36:27.291582 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:27.291599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:27.291713 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:27.292024 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:27.292027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:27.292427 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:27.292566 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.292575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.293576 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.293579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.294487 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.294497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.295164 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.295168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.295333 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.295892 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:27.295909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:27.299083 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":271,"threads_started":1}
W20260929 16:36:27.299538 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:27.291254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:33635 (local address 127.17.205.62:42549)
0929 16:36:27.291475 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:27.291481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:27.291673 (+ 192us) server_negotiation.cc:407] Connection header received
0929 16:36:27.291813 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:27.291817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:27.291896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:27.291976 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:27.292685 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.293443 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.294628 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.295051 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.295155 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.299319 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.3:33635: BlockingRecv error: recv got EOF from 127.17.205.3:33635 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.1@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:27.467919 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:27.459296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.1:43841)
0929 16:36:27.459799 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:27.459821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:27.459933 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:27.460213 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:27.460216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:27.460544 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:27.460715 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.460724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.461700 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.461704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.462611 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.462622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.463497 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.463503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.463639 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.464164 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:27.464180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:27.467656 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
W20260929 16:36:27.468087 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:27.459415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43841 (local address 127.17.205.62:42549)
0929 16:36:27.459665 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:27.459669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:27.459922 (+ 253us) server_negotiation.cc:407] Connection header received
0929 16:36:27.460028 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:27.460032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:27.460095 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:27.460175 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:27.460835 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.461577 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.462756 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.463271 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.463374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.467853 (+ 4479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.1:43841: BlockingRecv error: recv got EOF from 127.17.205.1:43841 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:36:27 dist-test-slave-3h01 krb5kdc[26464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.205.2@KRBTEST.COM for oryx/127.17.205.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:36:27.873060 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:27.864493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:42549 (local address 127.17.205.2:37019)
0929 16:36:27.864921 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:27.864936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:27.865057 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:27.865369 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:27.865374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:27.865738 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:27.865915 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.865927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.866901 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.866905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.867875 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:27.867889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.868549 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.868554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:27.868729 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.869299 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:27.869316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:27.872686 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.205.62:42549: Server oryx/127.17.205.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
W20260929 16:36:27.873131 27467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:27.864537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:37019 (local address 127.17.205.62:42549)
0929 16:36:27.864726 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:27.864733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:27.865018 (+ 285us) server_negotiation.cc:407] Connection header received
0929 16:36:27.865167 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:27.865173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:27.865234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:27.865312 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:27.866061 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.866760 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.868020 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:27.868413 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:27.868521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:27.872915 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.205.2:37019: BlockingRecv error: recv got EOF from 127.17.205.2:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:36:28.098536 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27051
I20260929 16:36:28.110688 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:28.099656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.17.205.1:45201)
0929 16:36:28.100448 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:28.100455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:28.100482 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:36:28.100546 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:28.100553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:28.100777 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:28.100952 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:28.102781 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.103889 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.105011 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.105541 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.105635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:28.107898 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:28.107907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:28.107920 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:28.107973 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:28.110181 (+ 2208us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:36:28.110467 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59618: Request ticket server kudu/127.17.205.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":524,"threads_started":1}
W20260929 16:36:28.110792 27515 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59618: Request ticket server kudu/127.17.205.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:36:28.110880 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27051: Not authorized: Client connection negotiation failed: client connection to 127.17.205.1:45201: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:36:28.111099 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27051
I20260929 16:36:28.124775 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27203
I20260929 16:36:28.136571 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:28.125888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.17.205.2:38469)
0929 16:36:28.126476 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:28.126483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:28.126513 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:36:28.126577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:28.126583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:28.126821 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:28.126985 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:28.128574 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.129606 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.130718 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.131308 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.131405 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:28.133606 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:28.133614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:28.133628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:28.133667 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:28.136042 (+ 2375us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:36:28.136313 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52102: Request ticket server kudu/127.17.205.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
W20260929 16:36:28.136613 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27203: Not authorized: Client connection negotiation failed: client connection to 127.17.205.2:38469: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:36:28.136667 27516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52102: Request ticket server kudu/127.17.205.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:36:28.136745 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27203
I20260929 16:36:28.150022 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27336
I20260929 16:36:28.161698 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:28.151035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.17.205.3:35005)
0929 16:36:28.151677 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:28.151687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:28.151717 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:36:28.151789 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:28.151797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:28.152166 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:28.152375 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:28.153580 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.154599 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.155724 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.156266 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.156362 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:28.158618 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:28.158626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:28.158641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:28.158678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:28.161020 (+ 2342us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:36:28.161361 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44576: Request ticket server kudu/127.17.205.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":304,"threads_started":1}
W20260929 16:36:28.161703 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27336: Not authorized: Client connection negotiation failed: client connection to 127.17.205.3:35005: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.205.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:36:28.161806 27517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44576: Request ticket server kudu/127.17.205.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:36:28.161854 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27336
I20260929 16:36:28.175230 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 26982
I20260929 16:36:28.176544 27044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:28.286787 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 26982
2026-09-29T16:36:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20864 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:28Z Disabled control of system clock
I20260929 16:36:28.355286 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35709
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:36:28.670595 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:28.670892 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:28.670967 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:28.678184 27522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:28.678294 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:28.678330 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:28.678383 27522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:28.678434 27522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:28.687642 27522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:35709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:28.688916 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:28.690625 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:28.702098 27527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:28.702350 27522 server_base.cc:1034] running on GCE node
W20260929 16:36:28.702011 27530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:28.702039 27528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:28.703480 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:28.704874 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:28.706069 27522 hybrid_clock.cc:648] HybridClock initialized: now 1790699788706019 us; error 39 us; skew 500 ppm
I20260929 16:36:28.706394 27522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:28.708526 27522 webserver.cc:460] Webserver started at http://127.17.205.62:36091/ using document root <none> and password file <none>
I20260929 16:36:28.709195 27522 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:28.709291 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:28.709600 27522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:28.712456 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "56ac46dfd7a44911b8f44c9cd126ceb3"
format_stamp: "Formatted at 2026-09-29 16:36:28 on dist-test-slave-3h01"
server_key: "dee52bfd50ea41bef90b40934b8577ed"
server_key_iv: "e62a5f031b3036d806fd7b87826a11c8"
server_key_version: "encryptionkey@0"
I20260929 16:36:28.713197 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "56ac46dfd7a44911b8f44c9cd126ceb3"
format_stamp: "Formatted at 2026-09-29 16:36:28 on dist-test-slave-3h01"
server_key: "dee52bfd50ea41bef90b40934b8577ed"
server_key_iv: "e62a5f031b3036d806fd7b87826a11c8"
server_key_version: "encryptionkey@0"
I20260929 16:36:28.718112 27522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:36:28.721351 27536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:28.722656 27522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:36:28.722810 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "56ac46dfd7a44911b8f44c9cd126ceb3"
format_stamp: "Formatted at 2026-09-29 16:36:28 on dist-test-slave-3h01"
server_key: "dee52bfd50ea41bef90b40934b8577ed"
server_key_iv: "e62a5f031b3036d806fd7b87826a11c8"
server_key_version: "encryptionkey@0"
I20260929 16:36:28.722992 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:28.761018 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:28.761924 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:28.762195 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:28.778052 27522 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:35709
I20260929 16:36:28.778059 27587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:35709 every 8 connection(s)
I20260929 16:36:28.779408 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:28.783392 27588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:28.785382 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27522
I20260929 16:36:28.785534 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:36:28.786161 18228 external_mini_cluster.cc:1428] Setting key f4cf01d77ac06b94d3216ab961af5dc7
I20260929 16:36:28.796386 27588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3: Bootstrap starting.
I20260929 16:36:28.800215 27588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:28.801517 27588 log.cc:824] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:28.801744 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:28.788259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.17.205.62:35709)
0929 16:36:28.789383 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:28.789399 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:28.789452 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:36:28.791665 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:28.791705 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:28.792371 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:28.792853 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:28.793431 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.795394 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.796685 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:28.797690 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:28.797865 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:28.798891 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:28.798901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:28.798919 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:28.799664 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:28.799763 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:28.799791 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:28.799904 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:28.800093 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:36:28.800438 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":325,"threads_started":1}
I20260929 16:36:28.804853 27588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3: No bootstrap required, opened a new log
I20260929 16:36:28.812158 27588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } }
I20260929 16:36:28.812521 27588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:28.812600 27588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ac46dfd7a44911b8f44c9cd126ceb3, State: Initialized, Role: FOLLOWER
I20260929 16:36:28.813166 27588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } }
I20260929 16:36:28.813369 27588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:28.813463 27588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:28.813598 27588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:28.815835 27588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } }
I20260929 16:36:28.816293 27588 leader_election.cc:302] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ac46dfd7a44911b8f44c9cd126ceb3; no voters:
I20260929 16:36:28.816816 27588 leader_election.cc:288] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:36:28.816905 27593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:28.818005 27593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [term 1 LEADER]: Becoming Leader. State: Replica: 56ac46dfd7a44911b8f44c9cd126ceb3, State: Running, Role: LEADER
I20260929 16:36:28.818495 27593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } }
I20260929 16:36:28.818928 27588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:28.821273 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ac46dfd7a44911b8f44c9cd126ceb3. Latest consensus state: current_term: 1 leader_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } } }
I20260929 16:36:28.821462 27594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ac46dfd7a44911b8f44c9cd126ceb3" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35709 } } }
I20260929 16:36:28.821694 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:28.821720 27594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:28.823592 27602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:28.829416 27602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:28.838568 27602 catalog_manager.cc:1347] Generated new cluster ID: 37fb4035c33246849dbf04ec144aaaae
I20260929 16:36:28.838698 27602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:28.864699 27602 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:36:28.865989 27602 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:28.876964 27602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56ac46dfd7a44911b8f44c9cd126ceb3: Generated new TSK 0
I20260929 16:36:28.877810 27602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:36:28.891108 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--builtin_ntp_servers=127.17.205.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:36:29.236892 27612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:29.237177 27612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:29.237294 27612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:29.244529 27612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:29.244633 27612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:29.244771 27612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:29.253751 27612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:29.255074 27612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:29.256944 27612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:29.269690 27620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:29.269804 27617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:29.269729 27618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:29.270623 27612 server_base.cc:1034] running on GCE node
I20260929 16:36:29.271191 27612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:29.272140 27612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:29.273334 27612 hybrid_clock.cc:648] HybridClock initialized: now 1790699789273265 us; error 65 us; skew 500 ppm
I20260929 16:36:29.273638 27612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:29.279691 27612 webserver.cc:460] Webserver started at http://127.17.205.1:46711/ using document root <none> and password file <none>
I20260929 16:36:29.280427 27612 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:29.280531 27612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:29.280830 27612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:29.283720 27612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "98df42ff7dcd45d59a229762b58442af"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "080d581c63369e033d48f590942812c1"
server_key_iv: "ba2d6264ef76d404b936204c5dd61ef5"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.284478 27612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "98df42ff7dcd45d59a229762b58442af"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "080d581c63369e033d48f590942812c1"
server_key_iv: "ba2d6264ef76d404b936204c5dd61ef5"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.289537 27612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:36:29.292958 27626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.294261 27612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:36:29.294415 27612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "98df42ff7dcd45d59a229762b58442af"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "080d581c63369e033d48f590942812c1"
server_key_iv: "ba2d6264ef76d404b936204c5dd61ef5"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.294620 27612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:29.328269 27612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:29.329144 27612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:29.329419 27612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:29.330377 27612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:29.331780 27612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:29.331872 27612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.331983 27612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:29.332050 27612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.361263 27612 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:44337
I20260929 16:36:29.361317 27738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:44337 every 8 connection(s)
I20260929 16:36:29.362571 27612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:29.366418 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27612
I20260929 16:36:29.366616 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:36:29.367296 18228 external_mini_cluster.cc:1428] Setting key 22277236491cb4291762dfbabe0238eb
I20260929 16:36:29.370142 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--builtin_ntp_servers=127.17.205.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:36:29.375687 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:29.365095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:37731 (local address 127.17.205.62:35709)
0929 16:36:29.365547 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:29.365552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:29.367319 (+ 1767us) server_negotiation.cc:407] Connection header received
0929 16:36:29.368641 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:29.368646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:29.368724 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:29.368815 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:29.370209 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.370949 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.373021 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.373406 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.373515 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:29.374955 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:29.374959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:29.374963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:29.374982 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:29.375023 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:29.375034 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:29.375100 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:29.375337 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:36:29.375416 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260929 16:36:29.377333 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:29.365569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35709 (local address 127.17.205.1:37731)
0929 16:36:29.366530 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:29.366589 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:29.368382 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:29.369074 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:29.369086 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:29.369211 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:29.370033 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:29.370059 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.371090 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.371096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:29.372893 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:29.372907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.373623 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.373629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:29.373903 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:29.374616 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:29.374619 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:29.374671 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:29.374770 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:29.375113 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:29.375122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:29.375129 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:36:29.375336 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 16:36:29.375758 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":281,"threads_started":1}
I20260929 16:36:29.378989 27739 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35709
I20260929 16:36:29.379329 27739 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:29.380252 27739 heartbeater.cc:511] Master 127.17.205.62:35709 requested a full tablet report, sending...
I20260929 16:36:29.382449 27553 ts_manager.cc:194] Registered new tserver with Master: 98df42ff7dcd45d59a229762b58442af (127.17.205.1:44337)
I20260929 16:36:29.383950 27553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.1:37731
W20260929 16:36:29.689544 27743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:29.689841 27743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:29.689958 27743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:29.697273 27743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:29.697388 27743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:29.697531 27743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:29.706764 27743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:29.708079 27743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:29.709831 27743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:29.722421 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:29.722481 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:29.722451 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:29.723122 27743 server_base.cc:1034] running on GCE node
I20260929 16:36:29.723755 27743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:29.724677 27743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:29.725868 27743 hybrid_clock.cc:648] HybridClock initialized: now 1790699789725803 us; error 63 us; skew 500 ppm
I20260929 16:36:29.726169 27743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:29.728416 27743 webserver.cc:460] Webserver started at http://127.17.205.2:39211/ using document root <none> and password file <none>
I20260929 16:36:29.729101 27743 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:29.729200 27743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:29.729506 27743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:29.732329 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "c0b1da8681924004aa12e63034d6ab90"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "15eed79ee75776ae6651b1271bd5cac1"
server_key_iv: "6049d5e73cee3ad9ba98a4adf46ba9ad"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.733112 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "c0b1da8681924004aa12e63034d6ab90"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "15eed79ee75776ae6651b1271bd5cac1"
server_key_iv: "6049d5e73cee3ad9ba98a4adf46ba9ad"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.738209 27743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:36:29.741781 27757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.743149 27743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:29.743356 27743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "c0b1da8681924004aa12e63034d6ab90"
format_stamp: "Formatted at 2026-09-29 16:36:29 on dist-test-slave-3h01"
server_key: "15eed79ee75776ae6651b1271bd5cac1"
server_key_iv: "6049d5e73cee3ad9ba98a4adf46ba9ad"
server_key_version: "encryptionkey@0"
I20260929 16:36:29.743568 27743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:29.768385 27743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:29.769292 27743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:29.769586 27743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:29.770537 27743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:29.771976 27743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:29.772069 27743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.772169 27743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:29.772236 27743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:29.802114 27743 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:46449
I20260929 16:36:29.802167 27869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:46449 every 8 connection(s)
I20260929 16:36:29.803360 27743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:29.804637 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27743
I20260929 16:36:29.804806 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:36:29.805253 18228 external_mini_cluster.cc:1428] Setting key 3fc4fdb4cd7d5c844c7b9b0d31ffe0eb
I20260929 16:36:29.808012 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--builtin_ntp_servers=127.17.205.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:36:29.815362 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:29.805843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:50919 (local address 127.17.205.62:35709)
0929 16:36:29.806031 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:29.806036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:29.807602 (+ 1566us) server_negotiation.cc:407] Connection header received
0929 16:36:29.809339 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:29.809350 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:29.809430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:29.809516 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:29.810572 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.811311 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.812802 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.813192 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.813298 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:29.814601 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:29.814607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:29.814612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:29.814634 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:29.814670 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:29.814684 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:29.814765 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:29.814935 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:36:29.815034 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:36:29.816397 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:29.806454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35709 (local address 127.17.205.2:50919)
0929 16:36:29.807459 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:29.807502 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:29.809083 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:29.809754 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:29.809762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:29.809859 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:29.810409 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:29.810424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.811461 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.811465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:29.812668 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:29.812679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:29.813403 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:29.813411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:29.813631 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:29.814276 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:29.814279 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:29.814318 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:29.814423 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:29.814765 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:29.814770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:29.814776 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:36:29.814909 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:36:29.815243 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":271,"threads_started":1}
I20260929 16:36:29.817754 27870 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35709
I20260929 16:36:29.818020 27870 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:29.818768 27870 heartbeater.cc:511] Master 127.17.205.62:35709 requested a full tablet report, sending...
I20260929 16:36:29.820204 27553 ts_manager.cc:194] Registered new tserver with Master: c0b1da8681924004aa12e63034d6ab90 (127.17.205.2:46449)
I20260929 16:36:29.820966 27553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.2:50919
W20260929 16:36:30.121541 27873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:30.121831 27873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:30.121943 27873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:30.129325 27873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:30.129446 27873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:30.129590 27873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:30.138736 27873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:30.140079 27873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:30.141868 27873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:30.154457 27878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:30.154479 27881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:30.154419 27879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:30.155575 27873 server_base.cc:1034] running on GCE node
I20260929 16:36:30.156159 27873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:30.157078 27873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:30.158280 27873 hybrid_clock.cc:648] HybridClock initialized: now 1790699790158223 us; error 47 us; skew 500 ppm
I20260929 16:36:30.158583 27873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:30.160765 27873 webserver.cc:460] Webserver started at http://127.17.205.3:33045/ using document root <none> and password file <none>
I20260929 16:36:30.161451 27873 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:30.161535 27873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:30.161784 27873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:30.164598 27873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "073c19a79a1148a8a8834759a8b4370f"
format_stamp: "Formatted at 2026-09-29 16:36:30 on dist-test-slave-3h01"
server_key: "ef9364b2aa7136c9e8351c0a1a508616"
server_key_iv: "b803c7dbfeb7f81ea19642752a8660b7"
server_key_version: "encryptionkey@0"
I20260929 16:36:30.165301 27873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "073c19a79a1148a8a8834759a8b4370f"
format_stamp: "Formatted at 2026-09-29 16:36:30 on dist-test-slave-3h01"
server_key: "ef9364b2aa7136c9e8351c0a1a508616"
server_key_iv: "b803c7dbfeb7f81ea19642752a8660b7"
server_key_version: "encryptionkey@0"
I20260929 16:36:30.170421 27873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:36:30.173942 27887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:30.175422 27873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:36:30.175566 27873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "073c19a79a1148a8a8834759a8b4370f"
format_stamp: "Formatted at 2026-09-29 16:36:30 on dist-test-slave-3h01"
server_key: "ef9364b2aa7136c9e8351c0a1a508616"
server_key_iv: "b803c7dbfeb7f81ea19642752a8660b7"
server_key_version: "encryptionkey@0"
I20260929 16:36:30.175748 27873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:30.201185 27873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:30.202025 27873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:30.202276 27873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:30.203199 27873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:30.204660 27873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:30.204743 27873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:30.204823 27873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:30.204874 27873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:30.234030 27873 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:40889
I20260929 16:36:30.234098 27999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:40889 every 8 connection(s)
I20260929 16:36:30.235351 27873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:30.239051 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 27873
I20260929 16:36:30.239235 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:36:30.239750 18228 external_mini_cluster.cc:1428] Setting key c5b94e98805b1ce3c21f3620307aac3c
I20260929 16:36:30.247197 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:30.237781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:46353 (local address 127.17.205.62:35709)
0929 16:36:30.237974 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:30.237980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:30.239484 (+ 1504us) server_negotiation.cc:407] Connection header received
0929 16:36:30.241233 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:30.241238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:30.241318 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:30.241408 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:30.242485 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.243197 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.244720 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.245147 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.245250 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:30.246514 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:30.246519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:30.246522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:30.246540 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:30.246569 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:30.246580 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:30.246650 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:30.246862 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:36:30.246971 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:36:30.248373 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:30.238305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35709 (local address 127.17.205.3:46353)
0929 16:36:30.239319 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:30.239364 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:30.240961 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:30.241647 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:30.241655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:30.241754 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:30.242332 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:30.242351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.243363 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.243367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:30.244591 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:30.244602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.245353 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.245360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:30.245575 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:30.246199 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:30.246201 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:30.246252 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:30.246332 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:30.246660 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:30.246672 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:30.246680 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:36:30.246850 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:36:30.247161 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":302,"threads_started":1}
I20260929 16:36:30.249666 28000 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35709
I20260929 16:36:30.249938 28000 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:30.250666 28000 heartbeater.cc:511] Master 127.17.205.62:35709 requested a full tablet report, sending...
I20260929 16:36:30.251991 27553 ts_manager.cc:194] Registered new tserver with Master: 073c19a79a1148a8a8834759a8b4370f (127.17.205.3:40889)
I20260929 16:36:30.252836 27553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.3:46353
I20260929 16:36:30.254662 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:36:30.260699 27741 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:30.259108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.17.205.62:35709)
0929 16:36:30.259311 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:30.259316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:30.259761 (+ 445us) server_negotiation.cc:407] Connection header received
0929 16:36:30.259912 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:30.259918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:30.259991 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:30.260069 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:30.260465 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49736: BlockingRecv error: recv got EOF from 127.0.0.1:49736 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:36:30.261358 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27612
I20260929 16:36:30.268601 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:30.262253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.17.205.1:44337)
0929 16:36:30.262758 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:30.262765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:30.262791 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:36:30.262845 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:30.262851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:30.263045 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:30.263167 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:30.263499 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.264465 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.265644 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.266203 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.266298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:30.267276 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:30.267285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:30.267298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:30.267881 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:30.267939 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:30.267959 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:30.268045 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:30.268141 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:36:30.268254 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":261,"threads_started":1}
I20260929 16:36:30.269775 27734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:30.349707 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27612
I20260929 16:36:30.362942 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27743
I20260929 16:36:30.370060 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:30.363869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.17.205.2:46449)
0929 16:36:30.364383 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:30.364390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:30.364418 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:36:30.364478 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:30.364484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:30.364702 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:30.364826 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:30.365141 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.366130 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.367296 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.367844 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.367949 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:30.368816 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:30.368824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:30.368840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:30.369413 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:30.369459 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:30.369478 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:30.369559 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:30.369665 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:36:30.369769 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260929 16:36:30.371253 27865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:30.450804 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27743
I20260929 16:36:30.464285 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27873
I20260929 16:36:30.471935 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:30.465131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.17.205.3:40889)
0929 16:36:30.465661 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:30.465669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:30.465698 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:36:30.465760 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:30.465766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:30.465983 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:30.466126 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:30.466570 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.467616 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.468823 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:30.469373 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:30.469496 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:30.470526 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:30.470534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:30.470547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:30.471132 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:30.471179 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:30.471199 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:30.471339 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:30.471469 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 16:36:30.471614 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":237,"threads_started":1}
I20260929 16:36:30.473176 27995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:30.552971 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27873
I20260929 16:36:30.565605 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 27522
I20260929 16:36:30.566670 27583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:30.665529 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 27522
2026-09-29T16:36:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2374 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:30Z Disabled control of system clock
I20260929 16:36:30.729997 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:39371
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:36:31.043408 28019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:31.043689 28019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:31.043808 28019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:31.050945 28019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:31.051059 28019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:31.051101 28019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:31.051179 28019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:31.051258 28019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:31.060484 28019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:39371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:31.061811 28019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:31.063661 28019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:31.075001 28024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.075065 28025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.075065 28027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:31.075663 28019 server_base.cc:1034] running on GCE node
I20260929 16:36:31.076393 28019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:31.077728 28019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:31.078925 28019 hybrid_clock.cc:648] HybridClock initialized: now 1790699791078867 us; error 51 us; skew 500 ppm
I20260929 16:36:31.079289 28019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:31.081450 28019 webserver.cc:460] Webserver started at http://127.17.205.62:46289/ using document root <none> and password file <none>
I20260929 16:36:31.082135 28019 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:31.082231 28019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:31.082523 28019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:31.085314 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "6c751990a2014715ba8481a34bed37bf"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "2851ca18c6fe38afb3c1bb3c01557515"
server_key_iv: "8637c2ecc2aec4fa1bab7d5021b56dd1"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.086061 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "6c751990a2014715ba8481a34bed37bf"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "2851ca18c6fe38afb3c1bb3c01557515"
server_key_iv: "8637c2ecc2aec4fa1bab7d5021b56dd1"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.091135 28019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:36:31.094411 28033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:31.095743 28019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:36:31.095891 28019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6c751990a2014715ba8481a34bed37bf"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "2851ca18c6fe38afb3c1bb3c01557515"
server_key_iv: "8637c2ecc2aec4fa1bab7d5021b56dd1"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.096083 28019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:31.101800 28019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:31.102530 28019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:31.102788 28019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:31.118814 28019 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:39371
I20260929 16:36:31.118834 28084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:39371 every 8 connection(s)
I20260929 16:36:31.120168 28019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:31.123822 28085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:31.129328 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28019
I20260929 16:36:31.129519 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:36:31.130115 18228 external_mini_cluster.cc:1428] Setting key 027be032ecd4128599eb91162b7f5f3f
I20260929 16:36:31.135107 28085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf: Bootstrap starting.
I20260929 16:36:31.138974 28085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:31.139441 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:31.132145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.17.205.62:39371)
0929 16:36:31.133221 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:31.133235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:31.133285 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:36:31.135460 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:31.135488 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:31.136031 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:31.136448 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:31.137477 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:31.137486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:31.137503 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:31.137519 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:31.137590 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:31.137611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:31.137805 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:31.137923 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:36:31.138207 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":299,"threads_started":1}
I20260929 16:36:31.140213 28085 log.cc:824] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:31.143067 28085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf: No bootstrap required, opened a new log
I20260929 16:36:31.150373 28085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } }
I20260929 16:36:31.150759 28085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:31.150848 28085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c751990a2014715ba8481a34bed37bf, State: Initialized, Role: FOLLOWER
I20260929 16:36:31.151441 28085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } }
I20260929 16:36:31.151665 28085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:31.151778 28085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:31.151921 28085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:31.154142 28085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } }
I20260929 16:36:31.154654 28085 leader_election.cc:302] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c751990a2014715ba8481a34bed37bf; no voters:
I20260929 16:36:31.155268 28085 leader_election.cc:288] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:36:31.155367 28090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:31.156001 28090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [term 1 LEADER]: Becoming Leader. State: Replica: 6c751990a2014715ba8481a34bed37bf, State: Running, Role: LEADER
I20260929 16:36:31.156517 28090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } }
I20260929 16:36:31.157285 28085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:31.159584 28092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c751990a2014715ba8481a34bed37bf. Latest consensus state: current_term: 1 leader_uuid: "6c751990a2014715ba8481a34bed37bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } } }
I20260929 16:36:31.159922 28092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:31.159075 28091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c751990a2014715ba8481a34bed37bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c751990a2014715ba8481a34bed37bf" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39371 } } }
I20260929 16:36:31.160367 28091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:31.161916 28097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:31.167122 28097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:31.176681 28097 catalog_manager.cc:1347] Generated new cluster ID: 7f250f6d2b89431cb6f772b64604db9f
I20260929 16:36:31.176802 28097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:31.202702 28097 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:36:31.203234 28097 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:31.216185 28097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c751990a2014715ba8481a34bed37bf: Generated new TSK 0
I20260929 16:36:31.217290 28097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:36:31.221750 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--builtin_ntp_servers=127.17.205.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:36:31.541814 28109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:31.542092 28109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:31.542208 28109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:31.549273 28109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:31.549381 28109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:31.549535 28109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:31.558395 28109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:31.559712 28109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:31.561486 28109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:31.573980 28114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.573941 28115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.574007 28117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:31.574534 28109 server_base.cc:1034] running on GCE node
I20260929 16:36:31.575104 28109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:31.576020 28109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:31.577204 28109 hybrid_clock.cc:648] HybridClock initialized: now 1790699791577165 us; error 38 us; skew 500 ppm
I20260929 16:36:31.577486 28109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:31.579532 28109 webserver.cc:460] Webserver started at http://127.17.205.1:43347/ using document root <none> and password file <none>
I20260929 16:36:31.580166 28109 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:31.580251 28109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:31.580535 28109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:31.584446 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "8b166cf034184120afd946d8560d5613"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "bb023c5fa19e3cd2cf460717064c3344"
server_key_iv: "5f895edbc8a9cc64eea1a75981a8b0cf"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.585155 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "8b166cf034184120afd946d8560d5613"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "bb023c5fa19e3cd2cf460717064c3344"
server_key_iv: "5f895edbc8a9cc64eea1a75981a8b0cf"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.589933 28109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:36:31.593115 28123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:31.594405 28109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:36:31.594547 28109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "8b166cf034184120afd946d8560d5613"
format_stamp: "Formatted at 2026-09-29 16:36:31 on dist-test-slave-3h01"
server_key: "bb023c5fa19e3cd2cf460717064c3344"
server_key_iv: "5f895edbc8a9cc64eea1a75981a8b0cf"
server_key_version: "encryptionkey@0"
I20260929 16:36:31.594738 28109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:31.600147 28109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:31.600914 28109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:31.601161 28109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:31.602017 28109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:31.603381 28109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:31.603471 28109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:31.603580 28109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:31.603646 28109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:31.632582 28109 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:33243
I20260929 16:36:31.632635 28235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:33243 every 8 connection(s)
I20260929 16:36:31.633860 28109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:31.642305 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28109
I20260929 16:36:31.642468 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:36:31.643016 18228 external_mini_cluster.cc:1428] Setting key 912816758bb416f8e56c2d3d2c66196e
I20260929 16:36:31.645255 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:31.636318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:59137 (local address 127.17.205.62:39371)
0929 16:36:31.636529 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:31.636535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:31.637850 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 16:36:31.639600 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:31.639605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:31.639657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:31.639745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:31.644558 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:31.644564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:31.644568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:31.644588 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:31.644624 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:31.644640 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:31.644743 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:31.644957 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:36:31.645016 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:36:31.645752 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--builtin_ntp_servers=127.17.205.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:36:31.646211 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:31.636840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:39371 (local address 127.17.205.1:59137)
0929 16:36:31.637714 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:31.637749 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:31.639336 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:31.639926 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:31.639933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:31.640035 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:31.644197 (+ 4162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:31.644201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:31.644253 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:31.644373 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:31.644736 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:31.644747 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:31.644756 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:36:31.645010 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 16:36:31.645216 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":238,"threads_started":1}
I20260929 16:36:31.647606 28236 heartbeater.cc:347] Connected to a master server at 127.17.205.62:39371
I20260929 16:36:31.647882 28236 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:31.648437 28236 heartbeater.cc:511] Master 127.17.205.62:39371 requested a full tablet report, sending...
I20260929 16:36:31.650326 28050 ts_manager.cc:194] Registered new tserver with Master: 8b166cf034184120afd946d8560d5613 (127.17.205.1:33243)
W20260929 16:36:31.964998 28239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:31.965283 28239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:31.965394 28239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:31.972424 28239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:31.972539 28239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:31.972675 28239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:31.981564 28239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:31.982833 28239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:31.984623 28239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:31.997020 28245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.997157 28247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:31.997028 28244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:31.997845 28239 server_base.cc:1034] running on GCE node
I20260929 16:36:31.998406 28239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:31.999358 28239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:32.000551 28239 hybrid_clock.cc:648] HybridClock initialized: now 1790699792000508 us; error 57 us; skew 500 ppm
I20260929 16:36:32.000846 28239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:32.002974 28239 webserver.cc:460] Webserver started at http://127.17.205.2:36847/ using document root <none> and password file <none>
I20260929 16:36:32.003682 28239 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:32.003780 28239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:32.004082 28239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:32.006825 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "be91877a9deb4875bcd3decb0463725d"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "f010d2145e13d59b91694fd3128fb467"
server_key_iv: "d8d11f6adb466cdcc09d07a5a0e51795"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.007608 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "be91877a9deb4875bcd3decb0463725d"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "f010d2145e13d59b91694fd3128fb467"
server_key_iv: "d8d11f6adb466cdcc09d07a5a0e51795"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.012451 28239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:36:32.015764 28253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.017083 28239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:36:32.017235 28239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "be91877a9deb4875bcd3decb0463725d"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "f010d2145e13d59b91694fd3128fb467"
server_key_iv: "d8d11f6adb466cdcc09d07a5a0e51795"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.017443 28239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:32.023090 28239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:32.023944 28239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:32.024201 28239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:32.025054 28239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:32.026376 28239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:32.026467 28239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.026575 28239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:32.026643 28239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.055761 28239 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:43331
I20260929 16:36:32.055810 28365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:43331 every 8 connection(s)
I20260929 16:36:32.057173 28239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:32.064397 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28239
I20260929 16:36:32.064558 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:36:32.065095 18228 external_mini_cluster.cc:1428] Setting key da3af83e7439ffb1bb4365f938a59e4d
I20260929 16:36:32.065342 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.059770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:43063 (local address 127.17.205.62:39371)
0929 16:36:32.059928 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.059933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.061433 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 16:36:32.063154 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.063158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.063244 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.063331 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.064630 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:32.064636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:32.064640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:32.064660 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:32.064699 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:32.064715 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:32.064851 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:32.065000 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:36:32.065064 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:36:32.066238 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.060306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:39371 (local address 127.17.205.2:43063)
0929 16:36:32.061273 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:32.061310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:32.062911 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:32.063539 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:32.063548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:32.063645 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:32.064302 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:32.064305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:32.064345 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:32.064433 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:32.064810 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:32.064818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:32.064824 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:36:32.065044 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 16:36:32.065289 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":274,"threads_started":1}
I20260929 16:36:32.067602 28366 heartbeater.cc:347] Connected to a master server at 127.17.205.62:39371
I20260929 16:36:32.067886 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--builtin_ntp_servers=127.17.205.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:36:32.067924 28366 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:32.068586 28366 heartbeater.cc:511] Master 127.17.205.62:39371 requested a full tablet report, sending...
I20260929 16:36:32.070088 28050 ts_manager.cc:194] Registered new tserver with Master: be91877a9deb4875bcd3decb0463725d (127.17.205.2:43331)
W20260929 16:36:32.391743 28369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:32.392048 28369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:32.392187 28369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:32.399449 28369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:32.399559 28369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:32.399680 28369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:32.408943 28369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:32.410178 28369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:32.411922 28369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:32.427724 28375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:32.427726 28377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:32.427867 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:32.428810 28369 server_base.cc:1034] running on GCE node
I20260929 16:36:32.429399 28369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:32.430330 28369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:32.431525 28369 hybrid_clock.cc:648] HybridClock initialized: now 1790699792431477 us; error 46 us; skew 500 ppm
I20260929 16:36:32.431826 28369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:36:32.433912 28369 webserver.cc:460] Webserver started at http://127.17.205.3:39405/ using document root <none> and password file <none>
I20260929 16:36:32.434578 28369 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:32.434675 28369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:32.434964 28369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:32.438151 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "7212ea786ba644199fa1b18a5a87a67b"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "c28f285ea57023bf7880a081c1069e21"
server_key_iv: "fc73be1d265d90a804887925748a6d5b"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.439105 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "7212ea786ba644199fa1b18a5a87a67b"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "c28f285ea57023bf7880a081c1069e21"
server_key_iv: "fc73be1d265d90a804887925748a6d5b"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.444125 28369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260929 16:36:32.447359 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.448760 28369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:36:32.448913 28369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "7212ea786ba644199fa1b18a5a87a67b"
format_stamp: "Formatted at 2026-09-29 16:36:32 on dist-test-slave-3h01"
server_key: "c28f285ea57023bf7880a081c1069e21"
server_key_iv: "fc73be1d265d90a804887925748a6d5b"
server_key_version: "encryptionkey@0"
I20260929 16:36:32.449115 28369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:32.454752 28369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:32.455631 28369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:32.455873 28369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:32.456766 28369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:32.458151 28369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:32.458242 28369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.458346 28369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:32.458420 28369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:32.486878 28369 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:41605
I20260929 16:36:32.486924 28495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:41605 every 8 connection(s)
I20260929 16:36:32.488170 28369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:32.489194 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28369
I20260929 16:36:32.489373 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:36:32.489854 18228 external_mini_cluster.cc:1428] Setting key e8a502748f5a099552aa8aabeb2cb40b
I20260929 16:36:32.496246 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.490826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:40253 (local address 127.17.205.62:39371)
0929 16:36:32.491013 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.491020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.492673 (+ 1653us) server_negotiation.cc:407] Connection header received
0929 16:36:32.494302 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.494307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.494370 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.494454 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.495617 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:32.495622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:32.495628 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:32.495645 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:32.495676 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:32.495686 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:32.495776 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:32.495941 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 16:36:32.496000 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:36:32.497118 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.491480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:39371 (local address 127.17.205.3:40253)
0929 16:36:32.492546 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:32.492583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:32.494071 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:32.494673 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:32.494681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:32.494769 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:32.495350 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:36:32.495353 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:32.495395 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:36:32.495471 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:32.495762 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:32.495772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:32.495779 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:36:32.496021 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 16:36:32.496261 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260929 16:36:32.498262 28496 heartbeater.cc:347] Connected to a master server at 127.17.205.62:39371
I20260929 16:36:32.498525 28496 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:32.499086 28496 heartbeater.cc:511] Master 127.17.205.62:39371 requested a full tablet report, sending...
I20260929 16:36:32.500393 28050 ts_manager.cc:194] Registered new tserver with Master: 7212ea786ba644199fa1b18a5a87a67b (127.17.205.3:41605)
I20260929 16:36:32.504650 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:36:32.510080 28088 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:36:32.508682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.17.205.62:39371)
0929 16:36:32.508825 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.508829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.509264 (+ 435us) server_negotiation.cc:407] Connection header received
0929 16:36:32.509406 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.509409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.509455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.509543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.509880 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48548: BlockingRecv error: recv got EOF from 127.0.0.1:48548 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:36:32.510608 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28109
I20260929 16:36:32.513525 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.511470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.17.205.1:33243)
0929 16:36:32.511911 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.511917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.511940 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:36:32.511991 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.511997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.512187 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.512299 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.513005 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:32.513013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:32.513023 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:32.513040 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:32.513076 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:32.513089 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:32.513182 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:32.513232 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:36:32.513301 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":202,"threads_started":1}
I20260929 16:36:32.514691 28231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:32.593326 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28109
I20260929 16:36:32.606702 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28239
I20260929 16:36:32.609867 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.607577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.17.205.2:43331)
0929 16:36:32.608066 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.608074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.608102 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:36:32.608164 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.608170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.608392 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.608522 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.609335 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:32.609340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:32.609352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:32.609363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:32.609402 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:32.609418 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:32.609521 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:32.609566 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 16:36:32.609650 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":208,"threads_started":1}
I20260929 16:36:32.611073 28361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:32.691820 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28239
I20260929 16:36:32.705839 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28369
I20260929 16:36:32.709873 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:32.707018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.17.205.3:41605)
0929 16:36:32.707810 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:32.707821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:32.707860 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:36:32.707952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:32.707961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:32.708282 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:32.708437 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:32.709288 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:32.709294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:32.709305 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:36:32.709320 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:32.709361 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:36:32.709374 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:32.709496 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:32.709545 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 16:36:32.709624 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":431,"threads_started":1}
I20260929 16:36:32.711081 28491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:32.790671 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28369
I20260929 16:36:32.803273 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28019
I20260929 16:36:32.804423 28080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:32.903014 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28019
2026-09-29T16:36:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2237 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:36:32 dist-test-slave-3h01 krb5kdc[28514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:36:32 dist-test-slave-3h01 krb5kdc[28514](info): set up 2 sockets
Sep 29 16:36:32 dist-test-slave-3h01 krb5kdc[28514](info): commencing operation
krb5kdc: starting...
W20260929 16:36:33.911850 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:36:34 dist-test-slave-3h01 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:36:34.131239 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35597
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:35597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:34.453559 28527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:34.453830 28527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:34.453943 28527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:34.461374 28527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:36:34.461498 28527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:34.461573 28527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:34.461642 28527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:36:34.461694 28527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:36:34.471926 28527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:35597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:34.473264 28527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:34.474602 28527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:34.485862 28532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:34.485898 28533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:34.486081 28535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:34.486737 28527 server_base.cc:1034] running on GCE node
I20260929 16:36:34.487564 28527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:34.488914 28527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:34.490106 28527 hybrid_clock.cc:648] HybridClock initialized: now 1790699794490054 us; error 44 us; skew 500 ppm
I20260929 16:36:34.490404 28527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:34 dist-test-slave-3h01 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:34.497817 28527 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:36:34.499328 28527 webserver.cc:460] Webserver started at http://127.17.205.62:40989/ using document root <none> and password file <none>
I20260929 16:36:34.500051 28527 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:34.500156 28527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:34.500422 28527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:34.503551 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "ea90a899daf9465aa2ac7f354965b313"
format_stamp: "Formatted at 2026-09-29 16:36:34 on dist-test-slave-3h01"
server_key: "4f46ef0293187c7776faed4c29f93832"
server_key_iv: "9ba2ce912ed832a687f9aa1cbb6066ac"
server_key_version: "encryptionkey@0"
I20260929 16:36:34.504272 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "ea90a899daf9465aa2ac7f354965b313"
format_stamp: "Formatted at 2026-09-29 16:36:34 on dist-test-slave-3h01"
server_key: "4f46ef0293187c7776faed4c29f93832"
server_key_iv: "9ba2ce912ed832a687f9aa1cbb6066ac"
server_key_version: "encryptionkey@0"
I20260929 16:36:34.509356 28527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:36:34.512760 28542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:34.514093 28527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:36:34.514225 28527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "ea90a899daf9465aa2ac7f354965b313"
format_stamp: "Formatted at 2026-09-29 16:36:34 on dist-test-slave-3h01"
server_key: "4f46ef0293187c7776faed4c29f93832"
server_key_iv: "9ba2ce912ed832a687f9aa1cbb6066ac"
server_key_version: "encryptionkey@0"
I20260929 16:36:34.514376 28527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:34.554663 28527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:34.558753 28527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:34.559005 28527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:34.575721 28527 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:35597
I20260929 16:36:34.575752 28593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:35597 every 8 connection(s)
I20260929 16:36:34.577104 28527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:36:34.580744 28594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:34.582650 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28527
I20260929 16:36:34.582844 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:36:34.583503 18228 external_mini_cluster.cc:1428] Setting key 656cc528b932565d5cd0c76603d31218
I20260929 16:36:34.593070 28594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: Bootstrap starting.
I20260929 16:36:34.596812 28594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:34.598089 28594 log.cc:824] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:36:34 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:34.600920 28594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: No bootstrap required, opened a new log
I20260929 16:36:34.606262 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:34.585554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.17.205.62:35597)
0929 16:36:34.586555 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:34.586571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:34.586619 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:36:34.588577 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:34.588602 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:34.589005 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:34.589342 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:34.591070 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:34.592428 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:34.594092 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:34.594764 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:34.594902 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:34.599402 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:34.599410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:34.599427 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:34.599461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:34.602278 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:34.603044 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:34.603050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:34.603057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:34.603248 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:34.603618 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:34.603621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:34.603623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:34.604174 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:34.604397 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:34.604823 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 16:36:34.605184 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":415,"threads_started":1}
I20260929 16:36:34.609263 28594 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } }
I20260929 16:36:34.609618 28594 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:34.609704 28594 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea90a899daf9465aa2ac7f354965b313, State: Initialized, Role: FOLLOWER
I20260929 16:36:34.610219 28594 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } }
I20260929 16:36:34.610421 28594 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:36:34.610569 28594 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:36:34.610736 28594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:34.612870 28594 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } }
I20260929 16:36:34.613315 28594 leader_election.cc:302] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea90a899daf9465aa2ac7f354965b313; no voters:
I20260929 16:36:34.613837 28594 leader_election.cc:288] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:36:34.613963 28599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:34.615012 28599 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [term 1 LEADER]: Becoming Leader. State: Replica: ea90a899daf9465aa2ac7f354965b313, State: Running, Role: LEADER
I20260929 16:36:34.615567 28599 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } }
I20260929 16:36:34.615885 28594 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:36:34.618227 28601 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea90a899daf9465aa2ac7f354965b313. Latest consensus state: current_term: 1 leader_uuid: "ea90a899daf9465aa2ac7f354965b313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } } }
I20260929 16:36:34.618569 28601 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:34.619107 28600 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea90a899daf9465aa2ac7f354965b313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea90a899daf9465aa2ac7f354965b313" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35597 } } }
I20260929 16:36:34.619405 28600 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313 [sys.catalog]: This master's current role is: LEADER
I20260929 16:36:34.619465 28606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:36:34.624840 28606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:36:34.633148 28606 catalog_manager.cc:1347] Generated new cluster ID: 13f395d7f64d4aebb111d61c59a2d6d3
I20260929 16:36:34.633267 28606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:36:34.654038 28606 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:36:34.656009 28606 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:36:34.669541 28606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: Generated new TSK 0
I20260929 16:36:34.670626 28606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:36:34.725728 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:35.042649 28620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:35.042912 28620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:35.043030 28620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:35.050099 28620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:35.050205 28620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:35.050361 28620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:36:35.059871 28620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:35.061180 28620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:35.062496 28620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:35.075060 28626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:35.075093 28628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:35.075169 28625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:35.075766 28620 server_base.cc:1034] running on GCE node
I20260929 16:36:35.076428 28620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:35.077378 28620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:35.078619 28620 hybrid_clock.cc:648] HybridClock initialized: now 1790699795078565 us; error 54 us; skew 500 ppm
I20260929 16:36:35.078948 28620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:35 dist-test-slave-3h01 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699795, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:35.082095 28620 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:36:35.083488 28620 webserver.cc:460] Webserver started at http://127.17.205.1:44099/ using document root <none> and password file <none>
I20260929 16:36:35.084167 28620 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:35.084261 28620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:35.084515 28620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:35.087359 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "2dc0e0042c4b442d93dfea099b57a38c"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "28a1a808365452c0ac4afa1b83390376"
server_key_iv: "fa92c3014cfa960f9064d739ed21ae4d"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.088064 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "2dc0e0042c4b442d93dfea099b57a38c"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "28a1a808365452c0ac4afa1b83390376"
server_key_iv: "fa92c3014cfa960f9064d739ed21ae4d"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.092979 28620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:36:35.096257 28635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.097556 28620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:36:35.097685 28620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2dc0e0042c4b442d93dfea099b57a38c"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "28a1a808365452c0ac4afa1b83390376"
server_key_iv: "fa92c3014cfa960f9064d739ed21ae4d"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.097833 28620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:35.109131 28620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:35.112795 28620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:35.113039 28620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:35.113909 28620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:35.115247 28620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:35.115324 28620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.115417 28620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:35.115456 28620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.144637 28620 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:38239
I20260929 16:36:35.144671 28747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:38239 every 8 connection(s)
I20260929 16:36:35.145856 28620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:36:35.146714 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28620
I20260929 16:36:35.146927 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:36:35.147719 18228 external_mini_cluster.cc:1428] Setting key 028b82221c7e78ea8660d031a913295c
Sep 29 16:36:35 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699795, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:35.166663 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:35.148658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:40379 (local address 127.17.205.62:35597)
0929 16:36:35.149102 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:35.149108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:35.150272 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 16:36:35.152644 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:35.152649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:35.152714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:35.152802 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:35.154632 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.155386 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.157384 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.157766 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.157854 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:35.161373 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:35.161378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:35.161381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:35.161414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:35.163781 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:35.164605 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:35.164608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:35.164610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:35.164715 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:35.165107 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:35.165109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:35.165111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:35.165367 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:35.165478 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:35.166238 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 16:36:35.166397 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260929 16:36:35.168113 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:35.149286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35597 (local address 127.17.205.1:40379)
0929 16:36:35.150098 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:35.150154 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:35.152360 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:35.153058 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:35.153070 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:35.153721 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:35.154476 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:35.154501 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.155512 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.155517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:35.157265 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:35.157283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.157976 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.157983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:35.158206 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:35.158832 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:35.158854 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:35.161121 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:35.163918 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:35.163927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:35.163944 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:35.164493 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:35.164797 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:35.164801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:35.164803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:35.165019 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:35.165474 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:35.165481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:35.165892 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 16:36:35.166183 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 16:36:35.166709 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
I20260929 16:36:35.169855 28748 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35597
I20260929 16:36:35.170200 28748 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:35.171110 28748 heartbeater.cc:511] Master 127.17.205.62:35597 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:36:35.173313 28559 ts_manager.cc:194] Registered new tserver with Master: 2dc0e0042c4b442d93dfea099b57a38c (127.17.205.1:38239)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:36:35.174762 28559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:40379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:36:35.199572 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:35.516168 28754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:35.516458 28754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:35.516589 28754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:35.523885 28754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:35.524001 28754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:35.524153 28754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:36:35.533886 28754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:35.535140 28754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:35.536422 28754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:35.548799 28760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:35.548714 28759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:35.548771 28762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:35.549696 28754 server_base.cc:1034] running on GCE node
I20260929 16:36:35.550284 28754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:35.551178 28754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:35.552402 28754 hybrid_clock.cc:648] HybridClock initialized: now 1790699795552353 us; error 44 us; skew 500 ppm
I20260929 16:36:35.552701 28754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:35 dist-test-slave-3h01 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699795, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:35.555672 28754 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:36:35.557011 28754 webserver.cc:460] Webserver started at http://127.17.205.2:42233/ using document root <none> and password file <none>
I20260929 16:36:35.557677 28754 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:35.557771 28754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:35.558003 28754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:35.560907 28754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "d2d51110080949019065d5210d53663b"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "9910c806dbf32b7d409ba2e12a2c0c39"
server_key_iv: "e43736190ff7ca3463e424cfdcdcad22"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.561635 28754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "d2d51110080949019065d5210d53663b"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "9910c806dbf32b7d409ba2e12a2c0c39"
server_key_iv: "e43736190ff7ca3463e424cfdcdcad22"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.566699 28754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:36:35.570048 28769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.571439 28754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:36:35.571604 28754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "d2d51110080949019065d5210d53663b"
format_stamp: "Formatted at 2026-09-29 16:36:35 on dist-test-slave-3h01"
server_key: "9910c806dbf32b7d409ba2e12a2c0c39"
server_key_iv: "e43736190ff7ca3463e424cfdcdcad22"
server_key_version: "encryptionkey@0"
I20260929 16:36:35.571770 28754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:35.585517 28754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:35.589207 28754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:35.589457 28754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:35.590299 28754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:35.591542 28754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:35.591609 28754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.591728 28754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:35.591779 28754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:35.620304 28754 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45199
I20260929 16:36:35.620345 28881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45199 every 8 connection(s)
I20260929 16:36:35.621670 28754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:36:35.630218 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28754
I20260929 16:36:35.630393 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:36:35.630769 18228 external_mini_cluster.cc:1428] Setting key b33ae22cf1d901576ab188cb00062613
Sep 29 16:36:35 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699795, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:35.640424 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:35.623972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:40137 (local address 127.17.205.62:35597)
0929 16:36:35.624169 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:35.624176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:35.625337 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 16:36:35.627062 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:35.627067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:35.627147 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:35.627241 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:35.628712 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.629420 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.630980 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.631479 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.631594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:35.635158 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:35.635163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:35.635166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:35.635196 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:35.637514 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:35.638182 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:35.638186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:35.638189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:35.638310 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:35.638751 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:35.638755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:35.638757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:35.639074 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:35.639190 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:35.640027 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 16:36:35.640197 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:36:35.641808 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:35.624517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35597 (local address 127.17.205.2:40137)
0929 16:36:35.625188 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:35.625225 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:35.626761 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:35.627498 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:35.627507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:35.627988 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:35.628547 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:35.628564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.629566 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.629572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:35.630847 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:35.630862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:35.631701 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:35.631709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:35.631915 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:35.632572 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:35.632611 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:35.634953 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:35.637660 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:35.637669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:35.637687 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:35.638075 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:35.638415 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:35.638421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:35.638425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:35.638643 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:35.639313 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:35.639321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:35.639706 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 16:36:35.639948 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 16:36:35.640449 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":247,"threads_started":1}
I20260929 16:36:35.643117 28882 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35597
I20260929 16:36:35.643450 28882 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:35.644289 28882 heartbeater.cc:511] Master 127.17.205.62:35597 requested a full tablet report, sending...
I20260929 16:36:35.645704 28559 ts_manager.cc:194] Registered new tserver with Master: d2d51110080949019065d5210d53663b (127.17.205.2:45199)
I20260929 16:36:35.646457 28559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:40137
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:36:35.680271 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:36:36.000113 28887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:36:36.000404 28887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:36:36.000516 28887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:36:36.007817 28887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:36:36.007937 28887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:36:36.008090 28887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:36:36.017961 28887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:36:36.019317 28887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:36:36.020694 28887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:36:36.033372 28892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:36.033435 28895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:36:36.033412 28893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:36:36.034247 28887 server_base.cc:1034] running on GCE node
I20260929 16:36:36.034821 28887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:36:36.035802 28887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:36:36.036996 28887 hybrid_clock.cc:648] HybridClock initialized: now 1790699796036952 us; error 37 us; skew 500 ppm
I20260929 16:36:36.037293 28887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:36:36 dist-test-slave-3h01 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699796, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:36:36.040656 28887 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:36:36.042213 28887 webserver.cc:460] Webserver started at http://127.17.205.3:40823/ using document root <none> and password file <none>
I20260929 16:36:36.042881 28887 fs_manager.cc:359] Metadata directory not provided
I20260929 16:36:36.042980 28887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:36:36.043287 28887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:36:36.046221 28887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9"
format_stamp: "Formatted at 2026-09-29 16:36:36 on dist-test-slave-3h01"
server_key: "229fac0dd2c8696e01b6a567c144ff3a"
server_key_iv: "78bc6aadcd2a1d2601b522de75dddc5f"
server_key_version: "encryptionkey@0"
I20260929 16:36:36.046938 28887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9"
format_stamp: "Formatted at 2026-09-29 16:36:36 on dist-test-slave-3h01"
server_key: "229fac0dd2c8696e01b6a567c144ff3a"
server_key_iv: "78bc6aadcd2a1d2601b522de75dddc5f"
server_key_version: "encryptionkey@0"
I20260929 16:36:36.052117 28887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:36:36.055564 28902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:36.056995 28887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:36:36.057134 28887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9"
format_stamp: "Formatted at 2026-09-29 16:36:36 on dist-test-slave-3h01"
server_key: "229fac0dd2c8696e01b6a567c144ff3a"
server_key_iv: "78bc6aadcd2a1d2601b522de75dddc5f"
server_key_version: "encryptionkey@0"
I20260929 16:36:36.057297 28887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:36:36.069818 28887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:36:36.073727 28887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:36:36.073971 28887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:36:36.074848 28887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:36:36.076241 28887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:36:36.076333 28887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:36.076428 28887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:36:36.076478 28887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:36:36.106201 28887 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:44643
I20260929 16:36:36.106240 29014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:44643 every 8 connection(s)
I20260929 16:36:36.107429 28887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:36:36.110680 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 28887
I20260929 16:36:36.110886 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:36:36.111339 18228 external_mini_cluster.cc:1428] Setting key 08b58627f8e243442b9c8f4deb6ed510
Sep 29 16:36:36 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699796, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:36:36.127032 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.109959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:43439 (local address 127.17.205.62:35597)
0929 16:36:36.110170 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.110177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.111522 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 16:36:36.113652 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.113656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.113727 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.113816 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:36.115376 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.116150 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.117704 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.118156 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.118258 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.121597 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:36.121601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:36.121605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:36.121636 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:36.124001 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.124739 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.124743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.124745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.124863 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.125279 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.125283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.125285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.125635 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:36.125788 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.126481 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.126701 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:36:36.128055 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.110554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35597 (local address 127.17.205.3:43439)
0929 16:36:36.111337 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.111388 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.113348 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.114080 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.114093 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.114633 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:36:36.115192 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.115236 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.116284 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.116288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.117586 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.117601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.118337 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.118342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.118554 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.119145 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:36:36.119166 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:36:36.121379 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:36:36.124138 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:36.124145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:36.124159 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:36.124614 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:36.124974 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:36:36.124980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:36:36.124984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:36:36.125167 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:36:36.125764 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:36:36.125772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:36:36.126119 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.126350 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.126896 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":301,"threads_started":1}
I20260929 16:36:36.129300 29015 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35597
I20260929 16:36:36.129587 29015 heartbeater.cc:464] Registering TS with master...
I20260929 16:36:36.130272 29015 heartbeater.cc:511] Master 127.17.205.62:35597 requested a full tablet report, sending...
I20260929 16:36:36.131601 28559 ts_manager.cc:194] Registered new tserver with Master: 96ff3e0f6e0f455c96bae0bce5e40ce9 (127.17.205.3:44643)
I20260929 16:36:36.132359 28559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:43439
I20260929 16:36:36.138669 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:36:36.156296 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.143134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.17.205.62:35597)
0929 16:36:36.143371 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.143377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.143698 (+ 321us) server_negotiation.cc:407] Connection header received
0929 16:36:36.143829 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.143833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.143917 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.143997 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:36.145775 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.146506 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.148329 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.148767 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.148869 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.151316 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:36.151321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:36.151324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:36.151357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:36.153764 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.154453 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.154457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.154459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.154590 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.154923 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.154928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.154932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.155335 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:36.155551 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.155825 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.155992 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:36:36.160511 28559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:36:36.167765 28559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:36:36.177969 28748 heartbeater.cc:503] Master 127.17.205.62:35597 was elected leader, sending a full tablet report...
I20260929 16:36:36.193470 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.180012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45199 (local address 127.0.0.1:48358)
0929 16:36:36.183385 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.183427 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.183625 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.185054 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.185060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.185086 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.185376 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.185384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.187424 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.187430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.190096 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.190115 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.192343 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.192351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.192627 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.192695 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.192817 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.193047 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3083,"thread_start_us":2918,"threads_started":1}
I20260929 16:36:36.195000 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.183948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:44643 (local address 127.0.0.1:36302)
0929 16:36:36.185762 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.185776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.185882 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.186859 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.186864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.186894 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.187316 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.187329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.189503 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.189510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.191638 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.191652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.194082 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.194090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.194295 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.194333 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.194421 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.194587 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725,"thread_start_us":1627,"threads_started":1}
I20260929 16:36:36.195293 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.180170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.17.205.2:45199)
0929 16:36:36.184154 (+ 3984us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.184165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.184194 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:36:36.184288 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.184296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.184587 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.184800 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.185534 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.187253 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.190356 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.192078 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.194378 (+ 2300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.194654 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.194739 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.194949 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3811,"thread_start_us":3720,"threads_started":1}
I20260929 16:36:36.196031 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.184272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.17.205.3:44643)
0929 16:36:36.185049 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.185058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.186038 (+ 980us) server_negotiation.cc:407] Connection header received
0929 16:36:36.186145 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.186153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.186396 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.186606 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.187517 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.189276 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.191829 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.193488 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.193638 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.193921 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.195514 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.195709 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":442,"threads_started":1}
I20260929 16:36:36.200141 28950 tablet_service.cc:1467] Processing CreateTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc (DEFAULT_TABLE table=test-table [id=a479dc4f326243f18e405e299b6c2a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:36.200141 28817 tablet_service.cc:1467] Processing CreateTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc (DEFAULT_TABLE table=test-table [id=a479dc4f326243f18e405e299b6c2a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:36.201668 28817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94accf19e3b434abe8d1ecd7b10bfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:36.201673 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94accf19e3b434abe8d1ecd7b10bfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:36.201781 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.185916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38239 (local address 127.0.0.1:54812)
0929 16:36:36.193799 (+ 7883us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.193821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.193940 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.197141 (+ 3201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.197146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.197172 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.197327 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.197335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.198754 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.198759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.199917 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.199929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.201233 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.201237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.201359 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.201392 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.201456 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.201567 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7724,"thread_start_us":9986,"threads_started":1}
I20260929 16:36:36.202802 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.186156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.17.205.1:38239)
0929 16:36:36.196502 (+ 10346us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.196513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.196554 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:36:36.196653 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.196661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.196924 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.197470 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.197651 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.198614 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.200067 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.201046 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.202061 (+ 1015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.202284 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.202361 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.202510 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10156,"thread_start_us":10703,"threads_started":1}
I20260929 16:36:36.206147 28683 tablet_service.cc:1467] Processing CreateTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc (DEFAULT_TABLE table=test-table [id=a479dc4f326243f18e405e299b6c2a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:36:36.207427 28683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94accf19e3b434abe8d1ecd7b10bfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:36:36.217183 29033 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Bootstrap starting.
I20260929 16:36:36.216377 29034 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Bootstrap starting.
I20260929 16:36:36.221114 29034 tablet_bootstrap.cc:654] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:36.221632 29033 tablet_bootstrap.cc:654] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:36.222235 29034 log.cc:824] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:36.222906 29033 log.cc:824] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:36.223541 29035 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Bootstrap starting.
I20260929 16:36:36.225441 29034 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: No bootstrap required, opened a new log
I20260929 16:36:36.225718 29034 ts_tablet_manager.cc:1397] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:36:36.226507 29033 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: No bootstrap required, opened a new log
I20260929 16:36:36.226737 29033 ts_tablet_manager.cc:1397] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:36:36.226913 29035 tablet_bootstrap.cc:654] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Neither blocks nor log segments found. Creating new log.
I20260929 16:36:36.228101 29035 log.cc:824] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:36:36.230911 29035 tablet_bootstrap.cc:492] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: No bootstrap required, opened a new log
I20260929 16:36:36.231172 29035 ts_tablet_manager.cc:1397] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 16:36:36.234107 29034 raft_consensus.cc:348] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.234465 29034 raft_consensus.cc:374] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:36.234604 29034 raft_consensus.cc:729] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d51110080949019065d5210d53663b, State: Initialized, Role: FOLLOWER
I20260929 16:36:36.235178 29034 consensus_queue.cc:260] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.236291 28882 heartbeater.cc:503] Master 127.17.205.62:35597 was elected leader, sending a full tablet report...
I20260929 16:36:36.236786 29034 ts_tablet_manager.cc:1428] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:36:36.237344 29033 raft_consensus.cc:348] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.237818 29033 raft_consensus.cc:374] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:36.237948 29033 raft_consensus.cc:729] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ff3e0f6e0f455c96bae0bce5e40ce9, State: Initialized, Role: FOLLOWER
I20260929 16:36:36.238531 29033 consensus_queue.cc:260] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.239749 29015 heartbeater.cc:503] Master 127.17.205.62:35597 was elected leader, sending a full tablet report...
I20260929 16:36:36.240366 29033 ts_tablet_manager.cc:1428] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:36:36.240953 29035 raft_consensus.cc:348] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.241397 29035 raft_consensus.cc:374] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:36:36.241499 29035 raft_consensus.cc:729] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc0e0042c4b442d93dfea099b57a38c, State: Initialized, Role: FOLLOWER
I20260929 16:36:36.242169 29035 consensus_queue.cc:260] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.244450 29035 ts_tablet_manager.cc:1428] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260929 16:36:36.358968 29016 tablet.cc:2369] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:36:36.373728 28883 tablet.cc:2369] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:36:36.398164 28749 tablet.cc:2369] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:36:36.469779 29041 raft_consensus.cc:482] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:36.470072 29041 raft_consensus.cc:504] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.471875 29041 leader_election.cc:288] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d51110080949019065d5210d53663b (127.17.205.2:45199), 96ff3e0f6e0f455c96bae0bce5e40ce9 (127.17.205.3:44643)
I20260929 16:36:36.478901 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.472506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:45133 (local address 127.17.205.2:45199)
0929 16:36:36.472733 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.472739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.473059 (+ 320us) server_negotiation.cc:407] Connection header received
0929 16:36:36.473331 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.473337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.473421 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.473515 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.474120 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.475058 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.477037 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.478006 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.478124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.478176 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.478531 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.478654 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 16:36:36.478995 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.472255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45199 (local address 127.17.205.1:45133)
0929 16:36:36.472969 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.472986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.473126 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.473604 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.473609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.473641 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.473979 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.473991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.475176 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.475182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.476877 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.476889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.478205 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.478211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.478377 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.478411 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.478488 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.478636 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":3,"thread_start_us":422,"threads_started":1}
I20260929 16:36:36.478955 29039 raft_consensus.cc:482] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:36:36.479620 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.473283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:51947 (local address 127.17.205.3:44643)
0929 16:36:36.473506 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.473513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.473887 (+ 374us) server_negotiation.cc:407] Connection header received
0929 16:36:36.474030 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.474035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.474111 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.474175 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.474805 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.475777 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.477547 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.478397 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.478542 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.478607 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.479262 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.479388 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:36:36.479648 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.473183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:44643 (local address 127.17.205.1:51947)
0929 16:36:36.473787 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.473801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.473906 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.474212 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.474216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.474240 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.474593 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.474604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.475892 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.475895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.477400 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.477414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.478840 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.478849 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.479032 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.479068 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.479189 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.479337 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":274,"threads_started":1}
I20260929 16:36:36.479610 29039 raft_consensus.cc:504] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.480293 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "2dc0e0042c4b442d93dfea099b57a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d51110080949019065d5210d53663b" is_pre_election: true
I20260929 16:36:36.480693 28837 raft_consensus.cc:2454] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc0e0042c4b442d93dfea099b57a38c in term 0.
I20260929 16:36:36.480712 28970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "2dc0e0042c4b442d93dfea099b57a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" is_pre_election: true
I20260929 16:36:36.481237 28970 raft_consensus.cc:2454] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc0e0042c4b442d93dfea099b57a38c in term 0.
I20260929 16:36:36.481359 28638 leader_election.cc:302] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc0e0042c4b442d93dfea099b57a38c, d2d51110080949019065d5210d53663b; no voters:
I20260929 16:36:36.481765 29039 leader_election.cc:288] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96ff3e0f6e0f455c96bae0bce5e40ce9 (127.17.205.3:44643), 2dc0e0042c4b442d93dfea099b57a38c (127.17.205.1:38239)
I20260929 16:36:36.482080 29041 raft_consensus.cc:2781] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:36:36.482200 29041 raft_consensus.cc:482] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:36:36.482306 29041 raft_consensus.cc:3037] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:36.485164 29041 raft_consensus.cc:504] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.485826 29041 leader_election.cc:288] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [CANDIDATE]: Term 1 election: Requested vote from peers d2d51110080949019065d5210d53663b (127.17.205.2:45199), 96ff3e0f6e0f455c96bae0bce5e40ce9 (127.17.205.3:44643)
I20260929 16:36:36.486188 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "2dc0e0042c4b442d93dfea099b57a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d51110080949019065d5210d53663b"
I20260929 16:36:36.486303 28970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "2dc0e0042c4b442d93dfea099b57a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9"
I20260929 16:36:36.486397 28837 raft_consensus.cc:3037] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:36.486495 28970 raft_consensus.cc:3037] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:36:36.489552 28837 raft_consensus.cc:2454] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc0e0042c4b442d93dfea099b57a38c in term 1.
I20260929 16:36:36.489635 28970 raft_consensus.cc:2454] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc0e0042c4b442d93dfea099b57a38c in term 1.
I20260929 16:36:36.490010 28638 leader_election.cc:302] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc0e0042c4b442d93dfea099b57a38c, 96ff3e0f6e0f455c96bae0bce5e40ce9; no voters:
I20260929 16:36:36.490103 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.481962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:47369 (local address 127.17.205.3:44643)
0929 16:36:36.482132 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.482137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.482690 (+ 553us) server_negotiation.cc:407] Connection header received
0929 16:36:36.482921 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.482926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.482991 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.483045 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.483748 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.484788 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.487581 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.488841 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.488967 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.489034 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.489729 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.489857 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:36:36.490087 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.482731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:59783 (local address 127.17.205.1:38239)
0929 16:36:36.482921 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.482927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.483459 (+ 532us) server_negotiation.cc:407] Connection header received
0929 16:36:36.483614 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.483623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.483688 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.483768 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:36:36.484440 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.485306 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.487728 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.488888 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.489002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.489066 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.489743 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.489867 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:36:36.490614 29041 raft_consensus.cc:2781] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:36:36.490664 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.482647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:38239 (local address 127.17.205.2:59783)
0929 16:36:36.483369 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.483383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.483504 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.483911 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.483916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.483938 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.484262 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.484273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.485463 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.485469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.487552 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.487570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.489261 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.489267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.489502 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.489594 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.490137 (+ 543us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.490264 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":93,"thread_start_us":426,"threads_started":1}
I20260929 16:36:36.490789 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.481944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:44643 (local address 127.17.205.2:47369)
0929 16:36:36.482602 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 16:36:36.482620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:36:36.482783 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:36:36.483236 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:36:36.483242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:36:36.483267 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:36:36.483618 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.483630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.484917 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.484923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.487382 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:36:36.487399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.489227 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.489233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:36:36.489486 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.489580 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:36:36.490292 (+ 712us) client_negotiation.cc:240] Negotiation successful
0929 16:36:36.490404 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":318,"thread_start_us":326,"threads_started":1}
I20260929 16:36:36.491084 29041 raft_consensus.cc:686] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 1 LEADER]: Becoming Leader. State: Replica: 2dc0e0042c4b442d93dfea099b57a38c, State: Running, Role: LEADER
I20260929 16:36:36.491554 28703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "d2d51110080949019065d5210d53663b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc0e0042c4b442d93dfea099b57a38c" is_pre_election: true
I20260929 16:36:36.491874 29041 consensus_queue.cc:237] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } }
I20260929 16:36:36.493842 28970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94accf19e3b434abe8d1ecd7b10bfbc" candidate_uuid: "d2d51110080949019065d5210d53663b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" is_pre_election: true
I20260929 16:36:36.494097 28970 raft_consensus.cc:2379] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d51110080949019065d5210d53663b in current term 1: Already voted for candidate 2dc0e0042c4b442d93dfea099b57a38c in this term.
I20260929 16:36:36.494670 28772 leader_election.cc:302] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d51110080949019065d5210d53663b; no voters: 2dc0e0042c4b442d93dfea099b57a38c, 96ff3e0f6e0f455c96bae0bce5e40ce9
I20260929 16:36:36.495074 29039 raft_consensus.cc:2726] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:36:36.496842 28559 catalog_manager.cc:5495] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dc0e0042c4b442d93dfea099b57a38c (127.17.205.1). New cstate: current_term: 1 leader_uuid: "2dc0e0042c4b442d93dfea099b57a38c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dc0e0042c4b442d93dfea099b57a38c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 38239 } health_report { overall_health: HEALTHY } } }
I20260929 16:36:36.554647 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.549987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.17.205.1:38239)
0929 16:36:36.550206 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.550210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.550231 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:36:36.550410 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.550413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.550487 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.550610 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:36:36.550884 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.551659 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.552845 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.553210 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.553296 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.553683 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:36:36.554163 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:36:36.554269 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.554304 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.554361 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 16:36:36.562382 28970 raft_consensus.cc:1261] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 1 FOLLOWER]: Refusing update from remote peer 2dc0e0042c4b442d93dfea099b57a38c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:36.562474 28837 raft_consensus.cc:1261] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 1 FOLLOWER]: Refusing update from remote peer 2dc0e0042c4b442d93dfea099b57a38c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:36:36.563195 29046 consensus_queue.cc:1035] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ff3e0f6e0f455c96bae0bce5e40ce9" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:36.563537 29041 consensus_queue.cc:1035] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d51110080949019065d5210d53663b" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:36:36.592836 28559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:36.593139 28559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:36:36.596750 28559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea90a899daf9465aa2ac7f354965b313: Sending DeleteTablet for 3 replicas of tablet b94accf19e3b434abe8d1ecd7b10bfbc
I20260929 16:36:36.597990 28817 tablet_service.cc:1514] Processing DeleteTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:36 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:48358
I20260929 16:36:36.598241 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:36 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:36302
I20260929 16:36:36.598923 28683 tablet_service.cc:1514] Processing DeleteTablet for tablet b94accf19e3b434abe8d1ecd7b10bfbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:36:36 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:54812
I20260929 16:36:36.599140 29060 tablet_replica.cc:330] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: stopping tablet replica
I20260929 16:36:36.599623 29060 raft_consensus.cc:2229] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:36.600188 29060 raft_consensus.cc:2258] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:36.600476 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28620
I20260929 16:36:36.607743 29061 tablet_replica.cc:330] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: stopping tablet replica
I20260929 16:36:36.607933 29059 tablet_replica.cc:330] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: stopping tablet replica
I20260929 16:36:36.608196 29061 raft_consensus.cc:2229] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:36:36.608263 29059 raft_consensus.cc:2229] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:36:36.608752 29059 raft_consensus.cc:2258] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:36.608894 29061 raft_consensus.cc:2258] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:36:36.610599 29060 ts_tablet_manager.cc:1905] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:36.610795 29061 ts_tablet_manager.cc:1905] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:36.610788 29059 ts_tablet_manager.cc:1905] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:36:36.617660 29061 ts_tablet_manager.cc:1918] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:36.617682 29059 ts_tablet_manager.cc:1918] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:36.617830 29061 log.cc:1196] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/b94accf19e3b434abe8d1ecd7b10bfbc
I20260929 16:36:36.617827 29059 log.cc:1196] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/b94accf19e3b434abe8d1ecd7b10bfbc
I20260929 16:36:36.618338 29061 ts_tablet_manager.cc:1939] T b94accf19e3b434abe8d1ecd7b10bfbc P 2dc0e0042c4b442d93dfea099b57a38c: Deleting consensus metadata
I20260929 16:36:36.618379 29059 ts_tablet_manager.cc:1939] T b94accf19e3b434abe8d1ecd7b10bfbc P d2d51110080949019065d5210d53663b: Deleting consensus metadata
I20260929 16:36:36.620172 28545 catalog_manager.cc:4841] TS d2d51110080949019065d5210d53663b (127.17.205.2:45199): tablet b94accf19e3b434abe8d1ecd7b10bfbc (table test-table [id=a479dc4f326243f18e405e299b6c2a41]) successfully deleted
I20260929 16:36:36.621042 28545 catalog_manager.cc:4841] TS 2dc0e0042c4b442d93dfea099b57a38c (127.17.205.1:38239): tablet b94accf19e3b434abe8d1ecd7b10bfbc (table test-table [id=a479dc4f326243f18e405e299b6c2a41]) successfully deleted
Sep 29 16:36:36 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:36:36.625582 29060 ts_tablet_manager.cc:1918] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:36:36.625814 29060 log.cc:1196] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/b94accf19e3b434abe8d1ecd7b10bfbc
I20260929 16:36:36.626320 29060 ts_tablet_manager.cc:1939] T b94accf19e3b434abe8d1ecd7b10bfbc P 96ff3e0f6e0f455c96bae0bce5e40ce9: Deleting consensus metadata
I20260929 16:36:36.628240 28545 catalog_manager.cc:4841] TS 96ff3e0f6e0f455c96bae0bce5e40ce9 (127.17.205.3:44643): tablet b94accf19e3b434abe8d1ecd7b10bfbc (table test-table [id=a479dc4f326243f18e405e299b6c2a41]) successfully deleted
I20260929 16:36:36.631850 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.601739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.17.205.1:38239)
0929 16:36:36.601927 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.601933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.607640 (+ 5707us) server_negotiation.cc:407] Connection header received
0929 16:36:36.607835 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.607841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.607919 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.608014 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:36.611777 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.612961 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.616453 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.617132 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.617235 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.624704 (+ 7469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:36.624720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:36.624737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:36.624786 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:36.628748 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.629618 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.629628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.629632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.629789 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.630169 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.630174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.630177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.630701 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:36.631031 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.631283 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.631517 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:36:36.632606 28743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:36.728631 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28620
I20260929 16:36:36.743106 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28754
Sep 29 16:36:36 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:36:36.757202 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.743885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.17.205.2:45199)
0929 16:36:36.744060 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.744064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.744108 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:36:36.744280 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.744284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.744346 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.744423 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:36.745693 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.746399 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.747659 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.748056 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.748148 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.752056 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:36.752070 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:36.752088 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:36.752126 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:36.754680 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.755394 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.755404 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.755407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.755564 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.755897 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.755900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.755902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.756272 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:36.756478 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.756707 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.756873 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:36:36.757843 28877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:36.845974 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28754
I20260929 16:36:36.859778 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28887
Sep 29 16:36:36 dist-test-slave-3h01 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:36:36.873310 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:36:36.860484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.17.205.3:44643)
0929 16:36:36.860657 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:36:36.860662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:36:36.860680 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:36:36.860833 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:36:36.860837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:36:36.860955 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:36:36.861038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:36:36.862303 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.862996 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.864182 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:36:36.864570 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:36:36.864657 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:36:36.868474 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:36:36.868489 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:36:36.868505 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:36:36.868537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:36:36.870996 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.871591 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.871597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.871600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.871716 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:36:36.872050 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:36:36.872054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:36:36.872056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:36:36.872446 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:36:36.872657 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:36:36.872852 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:36:36.873001 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:36:36.873899 29010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:36.961205 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28887
I20260929 16:36:36.974545 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 28527
I20260929 16:36:36.975661 28589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:36:37.080595 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 28527
2026-09-29T16:36:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4179 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:36:37 dist-test-slave-3h01 krb5kdc[29069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:36:37 dist-test-slave-3h01 krb5kdc[29069](info): set up 2 sockets
Sep 29 16:36:37 dist-test-slave-3h01 krb5kdc[29069](info): commencing operation
krb5kdc: starting...
W20260929 16:36:38.083823 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:36:38 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:36:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangeradmin_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangeradmin_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangerlookup_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangerlookup_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
I20260929 16:36:38.390029 18228 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaaWXO3/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/postgres -l logfile start
2026-09-29 16:36:40.639 UTC [29099] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:36:40.639 UTC [29099] LOG: listening on IPv4 address "127.17.205.20", port 45541
2026-09-29 16:36:40.642 UTC [29099] LOG: listening on Unix socket "/tmp/.s.PGSQL.45541"
2026-09-29 16:36:40.662 UTC [29102] LOG: database system was shut down at 2026-09-29 16:36:39 UTC
2026-09-29 16:36:40.665 UTC [29099] LOG: database system is ready to accept connections
I20260929 16:36:41.555478 18228 mini_postgres.cc:96] Postgres bound to 45541
2026-09-29 16:36:41.572 UTC [29110] FATAL: database "slave" does not exist
127.17.205.20:45541 - accepting connections
I20260929 16:36:41.573000 18228 mini_ranger.cc:161] Starting Ranger...
I20260929 16:36:41.598136 18228 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:36:42.036388 18228 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:36:42.036509 18228 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin
2026-09-29 16:36:42,317 [I] DB FLAVOR :POSTGRES
2026-09-29 16:36:42,318 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:36:42,318 [I] Checking connection..
2026-09-29 16:36:42,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:36:42,580 [I] Checking connection passed.
2026-09-29 16:36:42,580 [I] --------- Verifying version history table ---------
2026-09-29 16:36:42,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:36:42,844 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:36:42,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:36:43,100 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:36:43,100 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:36:43,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:36:43.343 UTC [29174] WARNING: there is no transaction in progress
2026-09-29 16:36:43,349 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:36:43,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:36:43,605 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:36:43,605 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:36:43,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:36:43,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:36:44,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h01.c.gcp-upstream.internal','N') ;"
2026-09-29 16:36:44,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:36:44,547 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:36:44,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:36:44,799 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:36:44,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:36:45,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:36:45,287 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:36:45,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:36:46.212 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.227 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.243 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.257 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.275 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.291 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.344 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.351 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.365 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.375 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.385 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.398 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.407 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.413 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.419 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46.425 UTC [29282] WARNING: there is no transaction in progress
2026-09-29 16:36:46,990 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:36:46,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:36:47,241 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:36:47,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:36:47,479 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:36:47,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:36:47,736 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:36:47,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:36:47,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:36:48,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h01.c.gcp-upstream.internal';"
2026-09-29 16:36:48,471 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:36:48,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:36:48,728 [I] DB_PATCHES have already been applied
I20260929 16:36:48.734144 18228 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaaWXO3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:36:48.734241 18228 mini_ranger.cc:193] Using host: 127.17.205.20
I20260929 16:36:48.748294 18228 mini_ranger.cc:227] Ranger admin URL: http://127.17.205.20:57371
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:36:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:36:49.814213 18228 mini_ranger.cc:160] Time spent starting Ranger: real 8.241s user 0.000s sys 0.040s
Sep 29, 2026 4:37:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:37:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 4:37:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:37:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:37:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:37:10.940454 18228 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangerkms_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/rangerkms_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.205.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.205.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:37:11 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699831, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:37:11.114158 18228 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:37:11.137310 18228 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:37:11.915267 18228 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:37:11.915437 18228 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms
I20260929 16:37:11.915571 18228 mini_ranger_kms.cc:227] Using postgres at 127.17.205.20:45541
2026-09-29 16:37:12,324 [I] DB FLAVOR :POSTGRES
2026-09-29 16:37:12,324 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:37:12,325 [I] Checking connection
2026-09-29 16:37:12,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:37:12,569 [I] connection success
2026-09-29 16:37:12,569 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:37:12,569 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:37:12,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:37:12,824 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:37:12,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:37:12,825 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:37:12,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:45541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:37:13,079 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:37:13.552712 18228 mini_ranger_kms.cc:310] Created kms service
I20260929 16:37:13.722240 18228 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:37:13.763671 18228 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:37:13 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699831, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:37:14.145236 18228 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:37:14.145565 18228 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:37:14.145617 18228 mini_ranger_kms.cc:240] Using host: 127.17.205.20
I20260929 16:37:14.165375 18228 mini_ranger_kms.cc:276] Ranger KMS PID: 29462
I20260929 16:37:14.165575 18228 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.205.20:37773
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:37:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:37:16.240037 18228 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.126s user 0.111s sys 0.328s
I20260929 16:37:16.240304 18228 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:37:16.240507 18228 mini_ranger_kms.cc:401] 127.17.205.20:37773/kms/v1/keys
Sep 29 16:37:16 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699836, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:37:16 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699836, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
Sep 29 16:37:17 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699836, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
Sep 29 16:37:19 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:37:19.474200 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44063
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:44063
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:37773/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:37:19.796018 29513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:37:19.796332 29513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:37:19.796445 29513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:37:19.803746 29513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:37:19.803874 29513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:37:19.803929 29513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:37:19.804006 29513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:37:19.804064 29513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:37:19.814323 29513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.205.20:37773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:44063
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:37:19.815716 29513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:37:19.817031 29513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:37:19.828534 29518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:19.828534 29519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:19.828603 29521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:37:19.829407 29513 server_base.cc:1034] running on GCE node
I20260929 16:37:19.830175 29513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:37:19.831686 29513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:37:19.832877 29513 hybrid_clock.cc:648] HybridClock initialized: now 1790699839832823 us; error 56 us; skew 500 ppm
I20260929 16:37:19.833204 29513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:37:19 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:37:19.842069 29513 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:37:19.843842 29513 webserver.cc:460] Webserver started at http://127.17.205.62:36189/ using document root <none> and password file <none>
I20260929 16:37:19.844656 29513 fs_manager.cc:359] Metadata directory not provided
I20260929 16:37:19.844761 29513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:37:19.845062 29513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:37:19 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:37:19.894754 29513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "8d737208c66c409ba742172ee70ac87e"
format_stamp: "Formatted at 2026-09-29 16:37:19 on dist-test-slave-3h01"
server_key: "3285335ea27b17b89f04321071872b03"
server_key_iv: "0d419484fd3a6008fe17359299cef9e4"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:19.895664 29513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "8d737208c66c409ba742172ee70ac87e"
format_stamp: "Formatted at 2026-09-29 16:37:19 on dist-test-slave-3h01"
server_key: "3285335ea27b17b89f04321071872b03"
server_key_iv: "0d419484fd3a6008fe17359299cef9e4"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:19.901297 29513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 16:37:19.920506 29528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:19.922500 29513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:37:19.922712 29513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "8d737208c66c409ba742172ee70ac87e"
format_stamp: "Formatted at 2026-09-29 16:37:19 on dist-test-slave-3h01"
server_key: "3285335ea27b17b89f04321071872b03"
server_key_iv: "0d419484fd3a6008fe17359299cef9e4"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:19.922974 29513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:37:19.938591 29513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:37:19.942226 29513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:37:19.942504 29513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:37:19.959357 29513 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:44063
I20260929 16:37:19.959389 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:44063 every 8 connection(s)
I20260929 16:37:19.960700 29513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:37:19.964527 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:37:19.969065 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 29513
I20260929 16:37:19.969262 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:37:19.970058 18228 external_mini_cluster.cc:1428] Setting key 18af197488513d92b52e183a5bad0129
I20260929 16:37:19.975976 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: Bootstrap starting.
I20260929 16:37:19.979576 29580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: Neither blocks nor log segments found. Creating new log.
I20260929 16:37:19.980682 29580 log.cc:824] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:37:19 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:37:19.983960 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: No bootstrap required, opened a new log
I20260929 16:37:19.991674 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:19.972133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.17.205.62:44063)
0929 16:37:19.972900 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:19.972913 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:19.972955 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:37:19.974631 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:19.974655 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:19.975100 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:19.975469 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:19.976922 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:19.978209 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:19.979904 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:19.980612 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:19.980758 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:19.984815 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:19.984823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:19.984838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:19.984877 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:19.987716 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:19.988372 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:19.988378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:19.988384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:19.988532 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:19.988893 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:19.988897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:19.988899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:19.989472 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:19.989789 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:19.990227 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 16:37:19.990619 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
I20260929 16:37:19.991528 29580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } }
I20260929 16:37:19.991914 29580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:37:19.992004 29580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d737208c66c409ba742172ee70ac87e, State: Initialized, Role: FOLLOWER
I20260929 16:37:19.992579 29580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } }
I20260929 16:37:19.992805 29580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:37:19.992926 29580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:37:19.993094 29580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:37:19.995488 29580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } }
I20260929 16:37:19.995955 29580 leader_election.cc:302] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d737208c66c409ba742172ee70ac87e; no voters:
I20260929 16:37:19.996464 29580 leader_election.cc:288] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:37:19.996585 29585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:37:19.997215 29585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [term 1 LEADER]: Becoming Leader. State: Replica: 8d737208c66c409ba742172ee70ac87e, State: Running, Role: LEADER
I20260929 16:37:19.997728 29585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } }
I20260929 16:37:19.998406 29580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:37:20.000510 29587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d737208c66c409ba742172ee70ac87e. Latest consensus state: current_term: 1 leader_uuid: "8d737208c66c409ba742172ee70ac87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } } }
I20260929 16:37:20.000957 29586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d737208c66c409ba742172ee70ac87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d737208c66c409ba742172ee70ac87e" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44063 } } }
I20260929 16:37:20.001297 29587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [sys.catalog]: This master's current role is: LEADER
I20260929 16:37:20.001382 29586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e [sys.catalog]: This master's current role is: LEADER
I20260929 16:37:20.011886 29593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:37:20.017283 29593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:37:20.017936 29580 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:37:20.039600 29593 catalog_manager.cc:1347] Generated new cluster ID: fe3a5c8c67cb495cbf6545600711af0e
I20260929 16:37:20.039871 29593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:37:20.054145 29593 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:37:20.056362 29593 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:37:20.072556 29593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: Generated new TSK 0
I20260929 16:37:20.073823 29593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:37:20.777091 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:37773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:37:21.178714 29625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:37:21.179018 29625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:37:21.179183 29625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:37:21.188030 29625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:37:21.188160 29625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:37:21.188309 29625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:37:21.198952 29625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.205.20:37773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:37:21.200455 29625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:37:21.202062 29625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:37:21.217121 29636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:21.217099 29635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:21.225402 29638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:37:21.225991 29625 server_base.cc:1034] running on GCE node
I20260929 16:37:21.226845 29625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:37:21.228802 29625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:37:21.230086 29625 hybrid_clock.cc:648] HybridClock initialized: now 1790699841229887 us; error 199 us; skew 500 ppm
I20260929 16:37:21.230432 29625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:37:21.233824 29625 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:37:21.235487 29625 webserver.cc:460] Webserver started at http://127.17.205.1:44185/ using document root <none> and password file <none>
I20260929 16:37:21.236212 29625 fs_manager.cc:359] Metadata directory not provided
I20260929 16:37:21.236346 29625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:37:21.236781 29625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:37:21.266723 29625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "67bbc19caab94417af9a5bb3cdfc9b7c"
format_stamp: "Formatted at 2026-09-29 16:37:21 on dist-test-slave-3h01"
server_key: "13a811cd6c7cbb43778136214b2ddd69"
server_key_iv: "95fcf08f099616a39a5e16e056d72578"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:21.267884 29625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "67bbc19caab94417af9a5bb3cdfc9b7c"
format_stamp: "Formatted at 2026-09-29 16:37:21 on dist-test-slave-3h01"
server_key: "13a811cd6c7cbb43778136214b2ddd69"
server_key_iv: "95fcf08f099616a39a5e16e056d72578"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:21.275609 29625 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260929 16:37:21.295790 29645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:21.297410 29625 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260929 16:37:21.297588 29625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "67bbc19caab94417af9a5bb3cdfc9b7c"
format_stamp: "Formatted at 2026-09-29 16:37:21 on dist-test-slave-3h01"
server_key: "13a811cd6c7cbb43778136214b2ddd69"
server_key_iv: "95fcf08f099616a39a5e16e056d72578"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:21.297812 29625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:37:21.332453 29625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:37:21.337917 29625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:37:21.338261 29625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:37:21.339504 29625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:37:21.341389 29625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:37:21.341486 29625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:21.341663 29625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:37:21.341741 29625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:21.377878 29625 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:34503
I20260929 16:37:21.377899 29757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:34503 every 8 connection(s)
I20260929 16:37:21.388988 29625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:37:21.394963 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 29625
I20260929 16:37:21.395193 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:37:21.396206 18228 external_mini_cluster.cc:1428] Setting key 39823be7465691695dab1c0b6107f743
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:37:21.447011 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:21.400350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:57693 (local address 127.17.205.62:44063)
0929 16:37:21.422672 (+ 22322us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:21.422677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:21.426463 (+ 3786us) server_negotiation.cc:407] Connection header received
0929 16:37:21.428554 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:21.428560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:21.428656 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:21.428760 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:21.431668 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:21.432647 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:21.434994 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:21.435904 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:21.436031 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:21.440641 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:21.440646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:21.440649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:21.440693 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:21.443428 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:21.444204 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:21.444209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:21.444213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:21.444371 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:21.444945 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:21.444950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:21.444953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:21.445288 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:21.445410 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:21.446429 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0929 16:37:21.446667 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22196,"thread_start_us":22883,"threads_started":1}
I20260929 16:37:21.449261 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:21.399761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44063 (local address 127.17.205.1:57693)
0929 16:37:21.425606 (+ 25845us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:21.425670 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:21.428150 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:21.429826 (+ 1676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:21.429839 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:21.430470 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:37:21.431431 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:21.431457 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:21.432836 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:21.432841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:21.434836 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:21.434850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:21.436513 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:21.436522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:21.436846 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:21.437663 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:37:21.437694 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:37:21.440411 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:37:21.443607 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:21.443617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:21.443635 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:21.444071 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:21.444500 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:21.444504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:21.444508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:21.444812 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:21.445502 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:37:21.445511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:37:21.445992 (+ 481us) client_negotiation.cc:769] Sending connection context
0929 16:37:21.447256 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0929 16:37:21.447833 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25449,"thread_start_us":28736,"threads_started":1}
I20260929 16:37:21.453320 29758 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44063
I20260929 16:37:21.453675 29758 heartbeater.cc:464] Registering TS with master...
I20260929 16:37:21.454558 29758 heartbeater.cc:511] Master 127.17.205.62:44063 requested a full tablet report, sending...
I20260929 16:37:21.456919 29545 ts_manager.cc:194] Registered new tserver with Master: 67bbc19caab94417af9a5bb3cdfc9b7c (127.17.205.1:34503)
I20260929 16:37:21.458789 29545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:57693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:37:21.477771 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:37773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
W20260929 16:37:21.879666 29764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:37:21.880187 29764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:37:21.880386 29764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:37:21.891093 29764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:37:21.891347 29764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:37:21.891611 29764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:37:21.906648 29764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.205.20:37773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:37:21.908775 29764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:37:21.910784 29764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:37:21.939549 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:21.944800 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:21.952214 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:37:21.953895 29764 server_base.cc:1034] running on GCE node
I20260929 16:37:21.954883 29764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:37:21.979573 29764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:37:21.981298 29764 hybrid_clock.cc:648] HybridClock initialized: now 1790699841981213 us; error 80 us; skew 500 ppm
I20260929 16:37:21.981824 29764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:37:21 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:37:21.986222 29764 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:37:22.007071 29764 webserver.cc:460] Webserver started at http://127.17.205.2:36489/ using document root <none> and password file <none>
I20260929 16:37:22.008312 29764 fs_manager.cc:359] Metadata directory not provided
I20260929 16:37:22.008572 29764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:37:22.009003 29764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:37:22 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:37:22.042824 29764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "d297a8a8ebe7424f9697261b3831aa98"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "33a36e196f51c1bf3e926995a9fba10f"
server_key_iv: "7100ac7b0075dd94cc195cea3a688237"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.044442 29764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "d297a8a8ebe7424f9697261b3831aa98"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "33a36e196f51c1bf3e926995a9fba10f"
server_key_iv: "7100ac7b0075dd94cc195cea3a688237"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.058709 29764 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.000s sys 0.011s
I20260929 16:37:22.094143 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.096549 29764 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.009s
I20260929 16:37:22.096832 29764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "d297a8a8ebe7424f9697261b3831aa98"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "33a36e196f51c1bf3e926995a9fba10f"
server_key_iv: "7100ac7b0075dd94cc195cea3a688237"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.097224 29764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:37:22.202935 29764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:37:22.208772 29764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:37:22.209095 29764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:37:22.210345 29764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:37:22.212296 29764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:37:22.212391 29764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.212567 29764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:37:22.212657 29764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.243489 29764 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45525
I20260929 16:37:22.243508 29900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45525 every 8 connection(s)
I20260929 16:37:22.245074 29764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:37:22.248337 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 29764
I20260929 16:37:22.248520 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:37:22.249068 18228 external_mini_cluster.cc:1428] Setting key 19894433457beb9514b843bf83d18b25
Sep 29 16:37:22 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699841, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:37:22.267273 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.247812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:36395 (local address 127.17.205.62:44063)
0929 16:37:22.251438 (+ 3626us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.251444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.251466 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:37:22.252999 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.253005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.253095 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.253195 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:22.254664 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.255479 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.257088 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.257479 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.257588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.260776 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:22.260781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:22.260784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:22.260835 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:22.263744 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.264662 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.264666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.264670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.264802 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.265302 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.265306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.265309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.265675 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:22.265805 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.266676 (+ 871us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.266892 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":3380,"threads_started":1}
I20260929 16:37:22.268563 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.248599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44063 (local address 127.17.205.2:36395)
0929 16:37:22.250652 (+ 2053us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:22.250719 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:22.252738 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:22.253458 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:22.253466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:22.253907 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:37:22.254503 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.254521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.255621 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.255631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.256958 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.256970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.257700 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.257708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.257922 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.258543 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:37:22.258566 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:37:22.260554 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:37:22.263890 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:22.263901 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:22.263921 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:22.264527 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:22.264907 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:22.264912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:22.264915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:22.265185 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:22.265805 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:37:22.265813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:37:22.266261 (+ 448us) client_negotiation.cc:769] Sending connection context
0929 16:37:22.266584 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 16:37:22.267121 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":1461,"threads_started":1}
I20260929 16:37:22.270067 29901 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44063
I20260929 16:37:22.270412 29901 heartbeater.cc:464] Registering TS with master...
I20260929 16:37:22.271309 29901 heartbeater.cc:511] Master 127.17.205.62:44063 requested a full tablet report, sending...
I20260929 16:37:22.273132 29545 ts_manager.cc:194] Registered new tserver with Master: d297a8a8ebe7424f9697261b3831aa98 (127.17.205.2:45525)
I20260929 16:37:22.274226 29545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:36395
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:37:22.304528 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:37773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:37:22.462699 29758 heartbeater.cc:503] Master 127.17.205.62:44063 was elected leader, sending a full tablet report...
W20260929 16:37:22.629503 29907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:37:22.629782 29907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:37:22.629899 29907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:37:22.637327 29907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:37:22.637444 29907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:37:22.637583 29907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:37:22.647670 29907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.205.20:37773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:37:22.648980 29907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:37:22.650314 29907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:37:22.663475 29913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:22.663498 29912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:37:22.664036 29915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:37:22.664618 29907 server_base.cc:1034] running on GCE node
I20260929 16:37:22.665232 29907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:37:22.666266 29907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:37:22.667455 29907 hybrid_clock.cc:648] HybridClock initialized: now 1790699842667401 us; error 51 us; skew 500 ppm
I20260929 16:37:22.667771 29907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:37:22 dist-test-slave-3h01 krb5kdc[29069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:37:22.670760 29907 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:37:22.672246 29907 webserver.cc:460] Webserver started at http://127.17.205.3:32885/ using document root <none> and password file <none>
I20260929 16:37:22.672930 29907 fs_manager.cc:359] Metadata directory not provided
I20260929 16:37:22.673046 29907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:37:22.673342 29907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:37:22 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:37:22.686640 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "4f714f9d6a384117a94804e0f29dd987"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "91c27239dca91eb183e4e0664b6c17f8"
server_key_iv: "eaf23a69e089939ee68742372c3f30eb"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.687530 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "4f714f9d6a384117a94804e0f29dd987"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "91c27239dca91eb183e4e0664b6c17f8"
server_key_iv: "eaf23a69e089939ee68742372c3f30eb"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.692880 29907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:37:22.708212 29922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.709774 29907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:37:22.709951 29907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "4f714f9d6a384117a94804e0f29dd987"
format_stamp: "Formatted at 2026-09-29 16:37:22 on dist-test-slave-3h01"
server_key: "91c27239dca91eb183e4e0664b6c17f8"
server_key_iv: "eaf23a69e089939ee68742372c3f30eb"
server_key_version: "kuduclusterkey@0"
I20260929 16:37:22.710173 29907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:37:22.744187 29907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:37:22.747987 29907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:37:22.748271 29907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:37:22.749225 29907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:37:22.750556 29907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:37:22.750630 29907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.750753 29907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:37:22.750803 29907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:37:22.779335 29907 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:37483
I20260929 16:37:22.779373 30034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:37483 every 8 connection(s)
I20260929 16:37:22.780584 29907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:37:22.788422 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 29907
I20260929 16:37:22.788597 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:37:22.788995 18228 external_mini_cluster.cc:1428] Setting key bbe85813f683349ba9ceca4c61463dd2
Sep 29 16:37:22 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:37:22.798545 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.782900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:40667 (local address 127.17.205.62:44063)
0929 16:37:22.783403 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.783409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.784341 (+ 932us) server_negotiation.cc:407] Connection header received
0929 16:37:22.785970 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.785974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.786054 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.786152 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:22.787552 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.788413 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.790071 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.790501 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.790622 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.793731 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:22.793736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:22.793739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:22.793773 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:22.796035 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.796675 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.796679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.796682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.796793 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.797130 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.797133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.797134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.797388 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:22.797491 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.798115 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.798291 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":222,"threads_started":1}
I20260929 16:37:22.799645 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.783501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44063 (local address 127.17.205.3:40667)
0929 16:37:22.784184 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:22.784228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:22.785734 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:22.786351 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:22.786359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:22.786762 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:37:22.787398 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.787418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.788548 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.788553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.789946 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.789957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.790713 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.790720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.790976 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.791623 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:37:22.791647 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:37:22.793539 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:37:22.796156 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:22.796163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:22.796178 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:22.796586 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:22.796866 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:37:22.796869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:37:22.796871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:37:22.797041 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:37:22.797494 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:37:22.797501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:37:22.797826 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 16:37:22.798049 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:37:22.798450 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":246,"threads_started":1}
I20260929 16:37:22.801054 30035 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44063
I20260929 16:37:22.801322 30035 heartbeater.cc:464] Registering TS with master...
I20260929 16:37:22.802116 30035 heartbeater.cc:511] Master 127.17.205.62:44063 requested a full tablet report, sending...
I20260929 16:37:22.803634 29545 ts_manager.cc:194] Registered new tserver with Master: 4f714f9d6a384117a94804e0f29dd987 (127.17.205.3:37483)
I20260929 16:37:22.804329 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:37:22.804632 29545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:40667
I20260929 16:37:22.821497 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.809173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.17.205.62:44063)
0929 16:37:22.809326 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.809330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.809674 (+ 344us) server_negotiation.cc:407] Connection header received
0929 16:37:22.809805 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.809809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.809878 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.809956 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:22.811333 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.812044 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.813601 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.814071 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.814172 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.816480 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:22.816485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:22.816489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:22.816525 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:22.818930 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.819654 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.819659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.819661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.819814 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:22.820171 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:22.820175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:22.820177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:22.820460 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:22.820688 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.820991 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.821200 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:37:22.825666 29545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:37:22.832800 29545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:37:22.858074 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.844859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:34503 (local address 127.0.0.1:51512)
0929 16:37:22.847438 (+ 2579us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:22.847458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:22.847748 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:22.848749 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:22.848757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:22.848789 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:37:22.849254 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.849266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.851432 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.851438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.854105 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.854544 (+ 439us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.856954 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.856961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.857203 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.857256 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:37:22.857409 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:37:22.857623 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2469,"mutex_wait_us":53,"thread_start_us":1529,"threads_started":1}
I20260929 16:37:22.859148 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.845373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.17.205.1:34503)
0929 16:37:22.846197 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.846207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.847721 (+ 1514us) server_negotiation.cc:407] Connection header received
0929 16:37:22.847950 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.847958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.848252 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.848430 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:37:22.849467 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.851234 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.854736 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.856379 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.856489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.856746 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.858583 (+ 1837us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.858785 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":454,"threads_started":1}
I20260929 16:37:22.859731 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.844583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37483 (local address 127.0.0.1:34806)
0929 16:37:22.846806 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:22.846846 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:22.847024 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:22.849368 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:22.849373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:22.849400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:37:22.849777 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.849788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.853318 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.853325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.857005 (+ 3680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.857021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.858855 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.858862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.859055 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.859103 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:37:22.859167 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:37:22.859316 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1955,"mutex_wait_us":1,"thread_start_us":2628,"threads_started":1}
I20260929 16:37:22.861694 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.844695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.17.205.3:37483)
0929 16:37:22.847635 (+ 2940us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.847645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.848785 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 16:37:22.848908 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.848915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.849158 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.849978 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:37:22.850252 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.853163 (+ 2911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.857206 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.858619 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.859951 (+ 1332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.860194 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.860277 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.860814 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2790,"thread_start_us":2630,"threads_started":1}
I20260929 16:37:22.863257 29693 tablet_service.cc:1467] Processing CreateTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 (DEFAULT_TABLE table=test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:37:22.864861 29693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d76f4e0c804073b7ce08dd2e19c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:37:22.865371 29970 tablet_service.cc:1467] Processing CreateTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 (DEFAULT_TABLE table=test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:37:22.866387 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.847808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.17.205.2:45525)
0929 16:37:22.860660 (+ 12852us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:22.860669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:22.860697 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:37:22.860796 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:22.860803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:22.861072 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:22.861283 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:37:22.861750 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.863061 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.864525 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.865455 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.865585 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.865796 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:22.865950 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:37:22.866107 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12677,"thread_start_us":12565,"threads_started":1}
I20260929 16:37:22.866715 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:22.847572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45525 (local address 127.0.0.1:54992)
0929 16:37:22.859495 (+ 11923us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:22.859512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:22.859612 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:22.861390 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:22.861394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:22.861420 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:37:22.861575 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.861584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.863189 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.863194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.864384 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:22.864402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:22.865610 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:22.865616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:22.865788 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:22.865821 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:37:22.865904 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:37:22.866043 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11822,"thread_start_us":12424,"threads_started":1}
I20260929 16:37:22.867583 29970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d76f4e0c804073b7ce08dd2e19c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:37:22.870393 29836 tablet_service.cc:1467] Processing CreateTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 (DEFAULT_TABLE table=test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:37:22.871944 29836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d76f4e0c804073b7ce08dd2e19c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:37:22.879767 30054 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Bootstrap starting.
I20260929 16:37:22.884255 30054 tablet_bootstrap.cc:654] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Neither blocks nor log segments found. Creating new log.
I20260929 16:37:22.885898 30054 log.cc:824] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:37:22.886399 30056 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Bootstrap starting.
I20260929 16:37:22.888062 30055 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Bootstrap starting.
I20260929 16:37:22.890483 30056 tablet_bootstrap.cc:654] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Neither blocks nor log segments found. Creating new log.
I20260929 16:37:22.891738 30055 tablet_bootstrap.cc:654] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Neither blocks nor log segments found. Creating new log.
I20260929 16:37:22.891880 30056 log.cc:824] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Log is configured to *not* fsync() on all Append() calls
I20260929 16:37:22.892086 30054 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: No bootstrap required, opened a new log
I20260929 16:37:22.892385 30054 ts_tablet_manager.cc:1397] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260929 16:37:22.892946 30055 log.cc:824] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Log is configured to *not* fsync() on all Append() calls
I20260929 16:37:22.895653 30056 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: No bootstrap required, opened a new log
I20260929 16:37:22.895911 30056 ts_tablet_manager.cc:1397] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:37:22.896121 30055 tablet_bootstrap.cc:492] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: No bootstrap required, opened a new log
I20260929 16:37:22.896395 30055 ts_tablet_manager.cc:1397] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260929 16:37:22.903761 30056 raft_consensus.cc:348] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.903770 30054 raft_consensus.cc:348] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.904168 30056 raft_consensus.cc:374] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:37:22.904284 30056 raft_consensus.cc:729] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d297a8a8ebe7424f9697261b3831aa98, State: Initialized, Role: FOLLOWER
I20260929 16:37:22.904300 30054 raft_consensus.cc:374] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:37:22.904402 30054 raft_consensus.cc:729] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bbc19caab94417af9a5bb3cdfc9b7c, State: Initialized, Role: FOLLOWER
I20260929 16:37:22.904918 30056 consensus_queue.cc:260] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.905033 30054 consensus_queue.cc:260] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.906033 29901 heartbeater.cc:503] Master 127.17.205.62:44063 was elected leader, sending a full tablet report...
I20260929 16:37:22.907021 30056 ts_tablet_manager.cc:1428] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:37:22.907739 30055 raft_consensus.cc:348] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.908249 30055 raft_consensus.cc:374] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:37:22.908380 30055 raft_consensus.cc:729] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f714f9d6a384117a94804e0f29dd987, State: Initialized, Role: FOLLOWER
I20260929 16:37:22.908789 30054 ts_tablet_manager.cc:1428] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260929 16:37:22.909166 30055 consensus_queue.cc:260] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:22.910921 30055 ts_tablet_manager.cc:1428] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 16:37:22.911284 30035 heartbeater.cc:503] Master 127.17.205.62:44063 was elected leader, sending a full tablet report...
W20260929 16:37:22.913190 29759 tablet.cc:2369] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:37:22.997184 29902 tablet.cc:2369] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:37:23.032104 30036 tablet.cc:2369] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:37:23.037439 30061 raft_consensus.cc:482] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:37:23.037715 30061 raft_consensus.cc:504] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:23.039352 30061 leader_election.cc:288] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f714f9d6a384117a94804e0f29dd987 (127.17.205.3:37483), d297a8a8ebe7424f9697261b3831aa98 (127.17.205.2:45525)
I20260929 16:37:23.045800 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.039646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:56839 (local address 127.17.205.3:37483)
0929 16:37:23.039865 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.039873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.040256 (+ 383us) server_negotiation.cc:407] Connection header received
0929 16:37:23.040435 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.040440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.040517 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.040601 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:37:23.041310 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.042197 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.044082 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.044891 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.045005 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.045066 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.045427 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.045558 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:37:23.045840 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.040499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:49521 (local address 127.17.205.2:45525)
0929 16:37:23.040708 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.040713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.040909 (+ 196us) server_negotiation.cc:407] Connection header received
0929 16:37:23.041073 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.041079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.041171 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.041267 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:37:23.041808 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.042576 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.044289 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.045038 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.045186 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.045253 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.045501 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.045625 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:37:23.045915 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.040257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45525 (local address 127.17.205.1:49521)
0929 16:37:23.040820 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:23.040838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:23.040948 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:23.041294 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:23.041298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:23.041322 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:37:23.041671 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:23.041683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.042712 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.042715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:23.044141 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:23.044156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.045232 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.045237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:23.045352 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.045380 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:37:23.045465 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:37:23.045589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260929 16:37:23.046099 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.039579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37483 (local address 127.17.205.1:56839)
0929 16:37:23.040147 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 16:37:23.040163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:37:23.040310 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:37:23.040758 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:37:23.040761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:37:23.040788 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:37:23.041182 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:23.041195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.042318 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.042323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:23.043929 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:37:23.043941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.045084 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.045090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:37:23.045281 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.045312 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:37:23.045381 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:37:23.045481 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"spinlock_wait_cycles":25472,"thread_start_us":274,"threads_started":1}
I20260929 16:37:23.046942 29856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d76f4e0c804073b7ce08dd2e19c782" candidate_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d297a8a8ebe7424f9697261b3831aa98" is_pre_election: true
I20260929 16:37:23.046943 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d76f4e0c804073b7ce08dd2e19c782" candidate_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f714f9d6a384117a94804e0f29dd987" is_pre_election: true
I20260929 16:37:23.047510 29990 raft_consensus.cc:2454] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bbc19caab94417af9a5bb3cdfc9b7c in term 0.
I20260929 16:37:23.047504 29856 raft_consensus.cc:2454] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bbc19caab94417af9a5bb3cdfc9b7c in term 0.
I20260929 16:37:23.048156 29646 leader_election.cc:302] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bbc19caab94417af9a5bb3cdfc9b7c, d297a8a8ebe7424f9697261b3831aa98; no voters:
I20260929 16:37:23.048494 30061 raft_consensus.cc:2781] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:37:23.048640 30061 raft_consensus.cc:482] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:37:23.048741 30061 raft_consensus.cc:3037] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:37:23.050846 30061 raft_consensus.cc:504] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:23.051438 30061 leader_election.cc:288] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [CANDIDATE]: Term 1 election: Requested vote from peers 4f714f9d6a384117a94804e0f29dd987 (127.17.205.3:37483), d297a8a8ebe7424f9697261b3831aa98 (127.17.205.2:45525)
I20260929 16:37:23.051831 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d76f4e0c804073b7ce08dd2e19c782" candidate_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f714f9d6a384117a94804e0f29dd987"
I20260929 16:37:23.051880 29856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d76f4e0c804073b7ce08dd2e19c782" candidate_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d297a8a8ebe7424f9697261b3831aa98"
I20260929 16:37:23.052016 29990 raft_consensus.cc:3037] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:37:23.052052 29856 raft_consensus.cc:3037] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:37:23.054148 29990 raft_consensus.cc:2454] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bbc19caab94417af9a5bb3cdfc9b7c in term 1.
I20260929 16:37:23.054163 29856 raft_consensus.cc:2454] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bbc19caab94417af9a5bb3cdfc9b7c in term 1.
I20260929 16:37:23.054528 29646 leader_election.cc:302] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bbc19caab94417af9a5bb3cdfc9b7c, d297a8a8ebe7424f9697261b3831aa98; no voters:
I20260929 16:37:23.054778 30061 raft_consensus.cc:2781] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:37:23.055176 30061 raft_consensus.cc:686] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 1 LEADER]: Becoming Leader. State: Replica: 67bbc19caab94417af9a5bb3cdfc9b7c, State: Running, Role: LEADER
I20260929 16:37:23.055733 30061 consensus_queue.cc:237] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } }
I20260929 16:37:23.060163 29543 catalog_manager.cc:5495] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bbc19caab94417af9a5bb3cdfc9b7c (127.17.205.1). New cstate: current_term: 1 leader_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67bbc19caab94417af9a5bb3cdfc9b7c" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34503 } health_report { overall_health: HEALTHY } } }
I20260929 16:37:23.093185 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.087892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.17.205.1:34503)
0929 16:37:23.088109 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.088115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.088173 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:37:23.088401 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.088405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.088484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.088619 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:37:23.088978 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.089715 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.091074 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.091510 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.091619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.092109 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:37:23.092617 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:37:23.092751 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.092848 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.092925 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:37:23.101058 29856 raft_consensus.cc:1261] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 1 FOLLOWER]: Refusing update from remote peer 67bbc19caab94417af9a5bb3cdfc9b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:37:23.101079 29990 raft_consensus.cc:1261] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 1 FOLLOWER]: Refusing update from remote peer 67bbc19caab94417af9a5bb3cdfc9b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:37:23.101956 30065 consensus_queue.cc:1035] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d297a8a8ebe7424f9697261b3831aa98" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:37:23.102295 30061 consensus_queue.cc:1035] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f714f9d6a384117a94804e0f29dd987" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:37:23.130059 29545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:37:23.130314 29545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:37:23.133904 29545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d737208c66c409ba742172ee70ac87e: Sending DeleteTablet for 3 replicas of tablet 57d76f4e0c804073b7ce08dd2e19c782
I20260929 16:37:23.135095 29970 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:37:23 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:34806
I20260929 16:37:23.135087 29836 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:37:23 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:54992
I20260929 16:37:23.135804 29693 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d76f4e0c804073b7ce08dd2e19c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:37:23 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:51512
I20260929 16:37:23.136207 30078 tablet_replica.cc:330] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: stopping tablet replica
I20260929 16:37:23.136689 30080 tablet_replica.cc:330] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: stopping tablet replica
I20260929 16:37:23.136659 30078 raft_consensus.cc:2229] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:37:23.137144 30080 raft_consensus.cc:2229] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:37:23.137171 30078 raft_consensus.cc:2258] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:37:23.137530 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 29625
I20260929 16:37:23.137799 30080 raft_consensus.cc:2258] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:37:23.140870 30078 ts_tablet_manager.cc:1905] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:37:23.141394 30080 ts_tablet_manager.cc:1905] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:37:23.147431 30078 ts_tablet_manager.cc:1918] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:37:23.147606 30078 log.cc:1196] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/57d76f4e0c804073b7ce08dd2e19c782
I20260929 16:37:23.147941 30080 ts_tablet_manager.cc:1918] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:37:23.148095 30078 ts_tablet_manager.cc:1939] T 57d76f4e0c804073b7ce08dd2e19c782 P 4f714f9d6a384117a94804e0f29dd987: Deleting consensus metadata
I20260929 16:37:23.148118 30080 log.cc:1196] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/57d76f4e0c804073b7ce08dd2e19c782
I20260929 16:37:23.148571 30080 ts_tablet_manager.cc:1939] T 57d76f4e0c804073b7ce08dd2e19c782 P 67bbc19caab94417af9a5bb3cdfc9b7c: Deleting consensus metadata
I20260929 16:37:23.149996 29529 catalog_manager.cc:4841] TS 4f714f9d6a384117a94804e0f29dd987 (127.17.205.3:37483): tablet 57d76f4e0c804073b7ce08dd2e19c782 (table test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]) successfully deleted
I20260929 16:37:23.150081 29532 catalog_manager.cc:4841] TS 67bbc19caab94417af9a5bb3cdfc9b7c (127.17.205.1:34503): tablet 57d76f4e0c804073b7ce08dd2e19c782 (table test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]) successfully deleted
I20260929 16:37:23.151672 30079 tablet_replica.cc:330] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: stopping tablet replica
I20260929 16:37:23.152043 30079 raft_consensus.cc:2229] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:37:23.152479 30079 raft_consensus.cc:2258] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:37:23.154106 30079 ts_tablet_manager.cc:1905] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:37:23.159413 30079 ts_tablet_manager.cc:1918] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:37:23.159574 30079 log.cc:1196] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/57d76f4e0c804073b7ce08dd2e19c782
I20260929 16:37:23.160068 30079 ts_tablet_manager.cc:1939] T 57d76f4e0c804073b7ce08dd2e19c782 P d297a8a8ebe7424f9697261b3831aa98: Deleting consensus metadata
I20260929 16:37:23.161635 29529 catalog_manager.cc:4841] TS d297a8a8ebe7424f9697261b3831aa98 (127.17.205.2:45525): tablet 57d76f4e0c804073b7ce08dd2e19c782 (table test-table [id=d8f7c9f7ae124781a40089a9c46a17bd]) successfully deleted
Sep 29 16:37:23 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:37:23.170598 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.138470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.17.205.1:34503)
0929 16:37:23.138662 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.138669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.153934 (+ 15265us) server_negotiation.cc:407] Connection header received
0929 16:37:23.154143 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.154148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.154230 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.154351 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:23.155996 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.157161 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.159109 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.159824 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.159961 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.164853 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:23.164864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:23.164878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:23.164918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:23.167911 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.168709 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.168716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.168718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.168865 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.169253 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.169256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.169257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.169618 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:23.169839 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.170153 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.170330 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:37:23.171362 29753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:37:23.267887 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 29625
I20260929 16:37:23.282886 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 29764
Sep 29 16:37:23 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:37:23.297569 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.283641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.17.205.2:45525)
0929 16:37:23.283798 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.283802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.283879 (+ 77us) server_negotiation.cc:407] Connection header received
0929 16:37:23.284050 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.284054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.284123 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.284197 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:23.285505 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.286240 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.287552 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.287931 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.288030 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.291991 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:23.292032 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:23.292054 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:23.292097 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:23.294851 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.295818 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.295825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.295827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.295955 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.296263 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.296266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.296268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.296616 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:23.296817 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.297084 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.297254 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:37:23.298215 29896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:37:23.385987 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 29764
I20260929 16:37:23.398762 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 29907
Sep 29 16:37:23 dist-test-slave-3h01 krb5kdc[29069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:37:23.413576 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:37:23.399484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.17.205.3:37483)
0929 16:37:23.399676 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:37:23.399682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:37:23.399718 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:37:23.399887 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:37:23.399893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:37:23.399960 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:37:23.400073 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:37:23.401432 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.402226 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.403632 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:37:23.404011 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:37:23.404128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:37:23.408112 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:37:23.408128 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:37:23.408144 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:37:23.408179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:37:23.410748 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.411562 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.411570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.411573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.411694 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:37:23.412061 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:37:23.412065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:37:23.412067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:37:23.412437 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:37:23.412700 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 16:37:23.413027 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:37:23.413239 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:37:23.414278 30030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:37:23.501685 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 29907
I20260929 16:37:23.514530 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 29513
I20260929 16:37:23.515739 29575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:37:23.623780 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 29513
I20260929 16:37:23.645123 18228 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:37:23.981099 18228 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:37:23.981319 18228 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:37:24.326359 18228 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:37:24.326 UTC [29099] LOG: received smart shutdown request
2026-09-29 16:37:24.329 UTC [29099] LOG: background worker "logical replication launcher" (PID 29108) exited with exit code 1
2026-09-29 16:37:24.330 UTC [29103] LOG: shutting down
2026-09-29 16:37:24.468 UTC [29099] LOG: database system is shut down
2026-09-29T16:37:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47499 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:37:24 dist-test-slave-3h01 krb5kdc[30094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:37:24 dist-test-slave-3h01 krb5kdc[30094](info): set up 2 sockets
Sep 29 16:37:24 dist-test-slave-3h01 krb5kdc[30094](info): commencing operation
krb5kdc: starting...
W20260929 16:37:25.599457 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:37:25 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:37:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangeradmin_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangeradmin_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangerlookup_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangerlookup_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
I20260929 16:37:25.895848 18228 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaaWXO3/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/postgres -l logfile start
2026-09-29 16:37:28.178 UTC [30124] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:37:28.178 UTC [30124] LOG: listening on IPv4 address "127.17.205.20", port 43007
2026-09-29 16:37:28.181 UTC [30124] LOG: listening on Unix socket "/tmp/.s.PGSQL.43007"
2026-09-29 16:37:28.202 UTC [30127] LOG: database system was shut down at 2026-09-29 16:37:26 UTC
2026-09-29 16:37:28.205 UTC [30124] LOG: database system is ready to accept connections
I20260929 16:37:29.095022 18228 mini_postgres.cc:96] Postgres bound to 43007
2026-09-29 16:37:29.111 UTC [30135] FATAL: database "slave" does not exist
127.17.205.20:43007 - accepting connections
I20260929 16:37:29.111699 18228 mini_ranger.cc:161] Starting Ranger...
I20260929 16:37:29.137238 18228 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:37:29.549481 18228 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:37:29.549579 18228 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin
2026-09-29 16:37:29,853 [I] DB FLAVOR :POSTGRES
2026-09-29 16:37:29,854 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:37:29,854 [I] Checking connection..
2026-09-29 16:37:29,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:37:30,092 [I] Checking connection passed.
2026-09-29 16:37:30,092 [I] --------- Verifying version history table ---------
2026-09-29 16:37:30,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:37:30,351 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:37:30,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:37:30,586 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:37:30,587 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:37:30,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:37:30.827 UTC [30199] WARNING: there is no transaction in progress
2026-09-29 16:37:30,833 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:37:30,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:37:31,062 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:37:31,062 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:37:31,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:37:31,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:37:31,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h01.c.gcp-upstream.internal','N') ;"
2026-09-29 16:37:31,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:37:32,067 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:37:32,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:37:32,323 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:37:32,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:37:32,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:37:32,811 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:37:32,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:37:33.713 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.729 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.744 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.759 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.777 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.795 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.856 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.862 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.875 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.885 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.895 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.907 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.917 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.923 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.929 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:33.935 UTC [30307] WARNING: there is no transaction in progress
2026-09-29 16:37:34,499 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:37:34,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:37:34,750 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:37:34,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:37:34,983 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:37:34,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:37:35,243 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:37:35,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:37:35,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:37:35,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h01.c.gcp-upstream.internal';"
2026-09-29 16:37:35,977 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:37:35,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:37:36,217 [I] DB_PATCHES have already been applied
I20260929 16:37:36.223167 18228 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaaWXO3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:37:36.223284 18228 mini_ranger.cc:193] Using host: 127.17.205.20
I20260929 16:37:36.236729 18228 mini_ranger.cc:227] Ranger admin URL: http://127.17.205.20:51807
Sep 29, 2026 4:37:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:37:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:37:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:37:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:37:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:37:37.285303 18228 mini_ranger.cc:160] Time spent starting Ranger: real 8.174s user 0.004s sys 0.033s
Sep 29, 2026 4:37:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:37:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 4:37:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:37:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:37:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:37:57.681646 18228 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.205.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.205.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangerkms_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.205.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/rangerkms_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.205.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.205.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.205.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/HTTP_127.17.205.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:37:57 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:37:57.844893 18228 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:37:57.870327 18228 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:37:58.571246 18228 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:37:58.571363 18228 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms
I20260929 16:37:58.571412 18228 mini_ranger_kms.cc:227] Using postgres at 127.17.205.20:43007
2026-09-29 16:37:58,985 [I] DB FLAVOR :POSTGRES
2026-09-29 16:37:58,985 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:37:58,985 [I] Checking connection
2026-09-29 16:37:58,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:37:59,250 [I] connection success
2026-09-29 16:37:59,250 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:37:59,250 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:37:59,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:37:59,545 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:37:59,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:37:59,546 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:37:59,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.205.20:43007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:37:59,874 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:38:00.331410 18228 mini_ranger_kms.cc:310] Created kms service
I20260929 16:38:00.553078 18228 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:38:00.593477 18228 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:38:00 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:00.989773 18228 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:38:00.990079 18228 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaaWXO3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaaWXO3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:38:00.990127 18228 mini_ranger_kms.cc:240] Using host: 127.17.205.20
I20260929 16:38:01.007431 18228 mini_ranger_kms.cc:276] Ranger KMS PID: 30487
I20260929 16:38:01.007539 18228 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.205.20:59149
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:38:03.094828 18228 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.250s user 0.111s sys 0.319s
I20260929 16:38:03.095049 18228 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:38:03.095201 18228 mini_ranger_kms.cc:401] 127.17.205.20:59149/kms/v1/keys
Sep 29 16:38:03 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:38:03 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
Sep 29 16:38:04 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.205.20@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
Sep 29 16:38:06 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:38:06.511065 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35785
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:35785
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:06.842381 30538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:06.842689 30538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:06.842806 30538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:06.850144 30538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:06.850265 30538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:06.850322 30538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:06.850399 30538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:06.850453 30538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:06.861159 30538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.205.20:59149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:35785
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:06.862548 30538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:06.864003 30538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:06.876220 30543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:06.876214 30544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:06.876257 30546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:06.877345 30538 server_base.cc:1034] running on GCE node
I20260929 16:38:06.878221 30538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:06.879769 30538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:06.881003 30538 hybrid_clock.cc:648] HybridClock initialized: now 1790699886880940 us; error 55 us; skew 500 ppm
I20260929 16:38:06.881384 30538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:06 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:06.884539 30538 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:06.886088 30538 webserver.cc:460] Webserver started at http://127.17.205.62:43021/ using document root <none> and password file <none>
I20260929 16:38:06.886750 30538 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:06.886861 30538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:06.887149 30538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:38:06 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:06.946360 30538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "a7a09f31796645d790947613b2a9df85"
format_stamp: "Formatted at 2026-09-29 16:38:06 on dist-test-slave-3h01"
server_key: "76ac4c750de24bca24351be0991cd698"
server_key_iv: "99185fc3f3540e2f11341d8d3086a8ab"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:06.947333 30538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "a7a09f31796645d790947613b2a9df85"
format_stamp: "Formatted at 2026-09-29 16:38:06 on dist-test-slave-3h01"
server_key: "76ac4c750de24bca24351be0991cd698"
server_key_iv: "99185fc3f3540e2f11341d8d3086a8ab"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:06.953083 30538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:38:06.973958 30553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:06.975669 30538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:38:06.975860 30538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a7a09f31796645d790947613b2a9df85"
format_stamp: "Formatted at 2026-09-29 16:38:06 on dist-test-slave-3h01"
server_key: "76ac4c750de24bca24351be0991cd698"
server_key_iv: "99185fc3f3540e2f11341d8d3086a8ab"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:06.976089 30538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:06.994568 30538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:06.998600 30538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:06.998906 30538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:07.016371 30538 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:35785
I20260929 16:38:07.016374 30604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:35785 every 8 connection(s)
I20260929 16:38:07.017777 30538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:07.021535 30605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:07.024266 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 30538
I20260929 16:38:07.024479 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:38:07.025280 18228 external_mini_cluster.cc:1428] Setting key 5c86665f27c861e00e1f31cab336fcb2
I20260929 16:38:07.034826 30605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Bootstrap starting.
I20260929 16:38:07.038345 30605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:07.039371 30605 log.cc:824] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:38:07 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:07.042761 30605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: No bootstrap required, opened a new log
I20260929 16:38:07.049441 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:07.027396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.17.205.62:35785)
0929 16:38:07.028186 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:07.028201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:07.028250 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:38:07.030482 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:07.030511 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:07.031050 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:07.031530 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:07.032668 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:07.034510 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:07.035894 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:07.036949 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:07.037126 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:07.041022 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:07.041032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:07.041051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:07.041095 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:07.044880 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:07.045586 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:07.045596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:07.045605 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:07.045797 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:07.046112 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:07.046117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:07.046123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:07.046853 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:07.047293 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:07.047582 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 16:38:07.048020 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":342,"threads_started":1}
I20260929 16:38:07.052290 30605 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:07.052605 30605 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:07.052671 30605 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a09f31796645d790947613b2a9df85, State: Initialized, Role: FOLLOWER
I20260929 16:38:07.053167 30605 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:07.053363 30605 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:07.053454 30605 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:07.053578 30605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:07.055670 30605 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:07.056133 30605 leader_election.cc:302] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a09f31796645d790947613b2a9df85; no voters:
I20260929 16:38:07.056684 30605 leader_election.cc:288] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:38:07.056933 30610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:07.057750 30610 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 LEADER]: Becoming Leader. State: Replica: a7a09f31796645d790947613b2a9df85, State: Running, Role: LEADER
I20260929 16:38:07.058208 30610 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:07.058619 30605 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:07.060941 30612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a09f31796645d790947613b2a9df85. Latest consensus state: current_term: 1 leader_uuid: "a7a09f31796645d790947613b2a9df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } } }
I20260929 16:38:07.060914 30611 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a09f31796645d790947613b2a9df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } } }
I20260929 16:38:07.061381 30611 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:07.061381 30612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:07.062119 30618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:07.067198 30618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:07.071729 30605 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:38:07.082676 30618 catalog_manager.cc:1347] Generated new cluster ID: a2d5560237c9463c941a06cdcf576388
I20260929 16:38:07.082831 30618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:07.114660 30618 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:38:07.116731 30618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:07.130775 30618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Generated new TSK 0
I20260929 16:38:07.131992 30618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:38:07.870285 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:38:08.289739 30650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:08.290056 30650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:08.290241 30650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:08.298128 30650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:08.298282 30650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:08.298476 30650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:08.310082 30650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:08.311789 30650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:08.313714 30650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:08.332471 30660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:08.332664 30661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:08.342621 30663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:08.342916 30650 server_base.cc:1034] running on GCE node
I20260929 16:38:08.343796 30650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:08.345125 30650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:08.346439 30650 hybrid_clock.cc:648] HybridClock initialized: now 1790699888346363 us; error 67 us; skew 500 ppm
I20260929 16:38:08.346982 30650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:08.351426 30650 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:08.356874 30650 webserver.cc:460] Webserver started at http://127.17.205.1:35831/ using document root <none> and password file <none>
I20260929 16:38:08.357952 30650 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:08.358098 30650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:08.358520 30650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:08.378579 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "a8772db5bd2a483696537a0b2e2c32d3"
format_stamp: "Formatted at 2026-09-29 16:38:08 on dist-test-slave-3h01"
server_key: "ed103c3ce2725508524226ed8cf7b3b0"
server_key_iv: "2c7803e9b206cf7364af42775e1e7a8c"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:08.379822 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "a8772db5bd2a483696537a0b2e2c32d3"
format_stamp: "Formatted at 2026-09-29 16:38:08 on dist-test-slave-3h01"
server_key: "ed103c3ce2725508524226ed8cf7b3b0"
server_key_iv: "2c7803e9b206cf7364af42775e1e7a8c"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:08.404207 30650 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.014s sys 0.009s
I20260929 16:38:08.445137 30670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:08.447434 30650 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.005s sys 0.015s
I20260929 16:38:08.447671 30650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "a8772db5bd2a483696537a0b2e2c32d3"
format_stamp: "Formatted at 2026-09-29 16:38:08 on dist-test-slave-3h01"
server_key: "ed103c3ce2725508524226ed8cf7b3b0"
server_key_iv: "2c7803e9b206cf7364af42775e1e7a8c"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:08.447961 30650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:08.490080 30650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:08.496084 30650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:08.496524 30650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:08.500761 30650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:08.502830 30650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:08.503050 30650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:38:08.503324 30650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:08.503455 30650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:08.812742 30650 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:40981
I20260929 16:38:08.812754 30782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:40981 every 8 connection(s)
I20260929 16:38:08.814189 30650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:08.821022 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 30650
I20260929 16:38:08.821228 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:38:08.821977 18228 external_mini_cluster.cc:1428] Setting key c73a1616c8587f2278680cc7a6dd999a
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:08.845348 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:08.818402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.1:56697)
0929 16:38:08.820830 (+ 2428us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:08.820889 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:08.824475 (+ 3586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:08.827635 (+ 3160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:08.827649 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:08.828325 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:08.829227 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:08.829256 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:08.830840 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:08.830846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:08.832500 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:08.832521 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:08.833322 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:08.833329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:08.833563 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:08.834228 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:08.834253 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:08.836307 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:08.840363 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:08.840373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:08.840395 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:08.841031 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:08.841510 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:08.841515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:08.841519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:08.841783 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:08.842528 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:08.842536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:08.842991 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 16:38:08.843328 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 16:38:08.843900 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"thread_start_us":1838,"threads_started":1}
I20260929 16:38:08.846053 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:08.817617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:56697 (local address 127.17.205.62:35785)
0929 16:38:08.824121 (+ 6504us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:08.824128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:08.824154 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:38:08.824793 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:08.824798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:08.824903 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:08.825003 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:08.829473 (+ 4470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:08.830653 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:08.832678 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:08.833085 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:08.833202 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:08.836601 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:08.836608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:08.836613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:08.836666 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:08.840153 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:08.841189 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:08.841195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:08.841200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:08.841375 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:08.841915 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:08.841920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:08.841923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:08.842291 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:08.842447 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:08.845408 (+ 2961us) server_negotiation.cc:299] Negotiation successful
0929 16:38:08.845674 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6351,"thread_start_us":7287,"threads_started":1}
I20260929 16:38:08.847813 30783 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:08.848171 30783 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:08.849069 30783 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:38:08.851709 30570 ts_manager.cc:194] Registered new tserver with Master: a8772db5bd2a483696537a0b2e2c32d3 (127.17.205.1:40981)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
I20260929 16:38:08.853858 30570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:56697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:38:08.890319 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:08 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699888, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
W20260929 16:38:09.352914 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:09.353230 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:09.353364 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:09.361078 30789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:09.361228 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:09.361393 30789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:09.371827 30789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:09.373178 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:09.374516 30789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:09.387681 30803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:09.387713 30806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:09.387755 30804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:09.389007 30789 server_base.cc:1034] running on GCE node
I20260929 16:38:09.389602 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:09.390566 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:09.391762 30789 hybrid_clock.cc:648] HybridClock initialized: now 1790699889391706 us; error 52 us; skew 500 ppm
I20260929 16:38:09.392089 30789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:09 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:09.395432 30789 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:09.397567 30789 webserver.cc:460] Webserver started at http://127.17.205.2:36043/ using document root <none> and password file <none>
I20260929 16:38:09.398526 30789 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:09.398686 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:09.399078 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:38:09 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:09.414850 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "1936341fc41343e0bd4500dc0bcfe318"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "414f4842fcc58282e9e36fdf97d0a0fe"
server_key_iv: "d1e8a3e492d58ee8cbb1207045276887"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:09.415803 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "1936341fc41343e0bd4500dc0bcfe318"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "414f4842fcc58282e9e36fdf97d0a0fe"
server_key_iv: "d1e8a3e492d58ee8cbb1207045276887"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:09.421303 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:38:09.438586 30813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:09.440335 30789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:38:09.440521 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "1936341fc41343e0bd4500dc0bcfe318"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "414f4842fcc58282e9e36fdf97d0a0fe"
server_key_iv: "d1e8a3e492d58ee8cbb1207045276887"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:09.440747 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:09.480371 30789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:09.484557 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:09.484846 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:09.485886 30789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:09.487349 30789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:09.487425 30789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:09.487529 30789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:09.487576 30789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:09.523180 30789 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45021
I20260929 16:38:09.523283 30925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45021 every 8 connection(s)
I20260929 16:38:09.524519 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:09.525004 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 30789
I20260929 16:38:09.525185 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:38:09.525835 18228 external_mini_cluster.cc:1428] Setting key 6b656268d6efa8a8c3c945f5bdfa8ad4
Sep 29 16:38:09 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:38:09.563761 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:09.528498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:46525 (local address 127.17.205.62:35785)
0929 16:38:09.544351 (+ 15853us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:09.544357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:09.544380 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:38:09.544456 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:09.544461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:09.544550 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:09.547030 (+ 2480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:09.547243 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:09.548396 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:09.550935 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:09.551576 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:09.551696 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:09.556356 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:09.556362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:09.556368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:09.556410 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:09.559937 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:09.560987 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:09.560993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:09.560997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:09.561173 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:09.561736 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:09.561740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:09.561742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:09.562086 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:09.562223 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:09.563052 (+ 829us) server_negotiation.cc:299] Negotiation successful
0929 16:38:09.563394 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15649,"thread_start_us":15577,"threads_started":1}
I20260929 16:38:09.564781 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:09.528275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.2:46525)
0929 16:38:09.540675 (+ 12400us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:09.540734 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:09.543359 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:09.545094 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:09.545155 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:09.545895 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:09.546817 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:09.546848 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:09.548599 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:09.548604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:09.550723 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:09.550739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:09.552008 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:09.552016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:09.552353 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:09.553165 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:09.553197 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:09.556064 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:09.560158 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:09.560169 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:09.560190 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:09.560839 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:09.561309 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:09.561314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:09.561318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:09.561622 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:09.562223 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:09.562232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:09.562605 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 16:38:09.562874 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 16:38:09.563483 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12049,"thread_start_us":10683,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:38:09.566326 30926 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:09.566625 30926 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:09.567389 30926 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:09.568817 30570 ts_manager.cc:194] Registered new tserver with Master: 1936341fc41343e0bd4500dc0bcfe318 (127.17.205.2:45021)
I20260929 16:38:09.569614 30570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:46525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:38:09.596692 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:09.857780 30783 heartbeater.cc:503] Master 127.17.205.62:35785 was elected leader, sending a full tablet report...
W20260929 16:38:09.941133 30932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:09.941413 30932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:09.941540 30932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:09.949070 30932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:09.949198 30932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:09.949345 30932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:09.959297 30932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:09.960562 30932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:09.961850 30932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:09.973773 30937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:09.974562 30938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:09.974975 30940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:09.975975 30932 server_base.cc:1034] running on GCE node
I20260929 16:38:09.976586 30932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:09.977548 30932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:09.978749 30932 hybrid_clock.cc:648] HybridClock initialized: now 1790699889978696 us; error 48 us; skew 500 ppm
I20260929 16:38:09.979080 30932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:09 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:09.982218 30932 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:09.983685 30932 webserver.cc:460] Webserver started at http://127.17.205.3:44971/ using document root <none> and password file <none>
I20260929 16:38:09.984383 30932 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:09.984490 30932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:09.984771 30932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:38:09 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:10.000941 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "d8f82a01cef14aa39df03ed3e68aefc1"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "6595dec294e2c0f0b1bfbcb674aaad18"
server_key_iv: "e4f1efb4c9e6e51da8902e2c8b0244b1"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:10.002075 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f82a01cef14aa39df03ed3e68aefc1"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "6595dec294e2c0f0b1bfbcb674aaad18"
server_key_iv: "e4f1efb4c9e6e51da8902e2c8b0244b1"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:10.009064 30932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:38:10.026753 30947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:10.028388 30932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:38:10.028556 30932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "d8f82a01cef14aa39df03ed3e68aefc1"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "6595dec294e2c0f0b1bfbcb674aaad18"
server_key_iv: "e4f1efb4c9e6e51da8902e2c8b0244b1"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:10.028765 30932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:10.066727 30932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:10.070843 30932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:10.071120 30932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:10.072184 30932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:10.073551 30932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:10.073627 30932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:10.073727 30932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:10.073783 30932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:10.103163 30932 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:46581
I20260929 16:38:10.103232 31059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:46581 every 8 connection(s)
I20260929 16:38:10.104478 30932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:10.112440 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 30932
I20260929 16:38:10.112591 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:38:10.113201 18228 external_mini_cluster.cc:1428] Setting key 4fbff4e8bec8eada9b95969c5e808732
Sep 29 16:38:10 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699889, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:10.123721 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:10.106902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45481 (local address 127.17.205.62:35785)
0929 16:38:10.107440 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:10.107446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:10.108486 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 16:38:10.110199 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:10.110204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:10.110276 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:10.110401 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:10.111838 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.112550 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.114641 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.115151 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.115304 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:10.118618 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:10.118625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:10.118630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:10.118668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:10.121033 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.121711 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.121715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.121717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.121855 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.122223 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.122225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.122227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.122503 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:10.122620 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:10.123290 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 16:38:10.123470 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":210,"threads_started":1}
I20260929 16:38:10.124743 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:10.107570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.3:45481)
0929 16:38:10.108318 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:10.108362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:10.109944 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:10.110591 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:10.110599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:10.111040 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:10.111682 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:10.111709 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.112716 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.112721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:10.114510 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:10.114524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.115396 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.115403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:10.115636 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:10.116308 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:10.116340 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:10.118366 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:10.121161 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:10.121170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:10.121186 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:10.121619 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:10.121933 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:10.121935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:10.121938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:10.122137 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:10.122611 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:10.122618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:10.122958 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 16:38:10.123193 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 16:38:10.123631 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260929 16:38:10.126185 31060 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:10.126492 31060 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:10.127251 31060 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:10.128830 30569 ts_manager.cc:194] Registered new tserver with Master: d8f82a01cef14aa39df03ed3e68aefc1 (127.17.205.3:46581)
I20260929 16:38:10.129730 30569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:45481
I20260929 16:38:10.141247 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:38:10.142930 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 30650
Sep 29 16:38:10 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:38:10.159297 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:10.143881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.17.205.1:40981)
0929 16:38:10.144464 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:10.144471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:10.144499 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:38:10.144558 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:10.144565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:10.144856 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:10.144995 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:10.146505 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.147602 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.149051 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.149624 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.149773 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:10.153812 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:10.153825 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:10.153840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:10.153881 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:10.156630 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.157364 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.157371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.157373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.157528 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.157823 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.157826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.157828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.158163 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:10.158394 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:10.158721 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:38:10.158928 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
I20260929 16:38:10.160657 30778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:10.242573 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 30650
I20260929 16:38:10.255455 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 30789
Sep 29 16:38:10 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:38:10.271241 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:10.256251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.17.205.2:45021)
0929 16:38:10.256810 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:10.256817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:10.256850 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:38:10.256911 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:10.256918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:10.257131 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:10.257257 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:10.258501 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.259593 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.261069 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.261621 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.261726 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:10.266222 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:10.266235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:10.266249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:10.266284 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:10.268964 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.269588 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.269594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.269597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.269719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.269987 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.269990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.269992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.270313 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:10.270533 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:10.270729 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 16:38:10.270924 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
I20260929 16:38:10.272434 30921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:10.353799 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 30789
I20260929 16:38:10.370113 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 30932
Sep 29 16:38:10 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:38:10.386006 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:10.371023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.17.205.3:46581)
0929 16:38:10.371609 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:10.371616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:10.371644 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:38:10.371702 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:10.371708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:10.371910 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:10.372032 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:10.373281 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.374311 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.375711 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:10.376304 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:10.376411 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:10.380675 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:10.380689 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:10.380705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:10.380745 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:10.383510 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.384209 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.384217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.384219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.384370 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:10.384687 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:10.384691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:10.384694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:10.385045 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:10.385286 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:10.385497 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 16:38:10.385718 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
I20260929 16:38:10.387323 31055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:10.467932 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 30932
I20260929 16:38:10.480744 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 30538
I20260929 16:38:10.481928 30600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:10.588002 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 30538
I20260929 16:38:10.605710 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35785
--webserver_interface=127.17.205.62
--webserver_port=43021
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:35785
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:10.922514 31072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:10.922796 31072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:10.922925 31072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:10.930171 31072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:10.930299 31072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:10.930378 31072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:10.930451 31072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:10.930516 31072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:10.940667 31072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:35785
--ranger_config_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=43021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:10.941933 31072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:10.943316 31072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:10.955053 31078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:10.955078 31077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:10.955703 31072 server_base.cc:1034] running on GCE node
W20260929 16:38:10.955737 31080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:10.956547 31072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:10.957983 31072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:10.959197 31072 hybrid_clock.cc:648] HybridClock initialized: now 1790699890959139 us; error 53 us; skew 500 ppm
I20260929 16:38:10.959578 31072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:10 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699890, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:10.962531 31072 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:10.964116 31072 webserver.cc:460] Webserver started at http://127.17.205.62:43021/ using document root <none> and password file <none>
I20260929 16:38:10.964787 31072 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:10.964905 31072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:38:11 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699890, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:11.028821 31072 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:38:11.031919 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:11.033514 31072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:38:11.033684 31072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "a7a09f31796645d790947613b2a9df85"
format_stamp: "Formatted at 2026-09-29 16:38:06 on dist-test-slave-3h01"
server_key: "76ac4c750de24bca24351be0991cd698"
server_key_iv: "99185fc3f3540e2f11341d8d3086a8ab"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:11.034667 31072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:11.051584 31072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:11.055300 31072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:11.055568 31072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:11.072360 31072 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:35785
I20260929 16:38:11.072381 31139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:35785 every 8 connection(s)
I20260929 16:38:11.073828 31072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:11.075099 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31072
I20260929 16:38:11.076155 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:40981
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:35785
--webserver_port=35831
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:11.081198 31140 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:38:11.085631 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Bootstrap starting.
I20260929 16:38:11.112465 31140 log.cc:824] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:11.123564 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:38:11.124188 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Bootstrap complete.
I20260929 16:38:11.137056 31140 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:11.138303 31140 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a09f31796645d790947613b2a9df85, State: Initialized, Role: FOLLOWER
I20260929 16:38:11.139114 31140 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:11.139464 31140 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:11.139623 31140 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:11.139824 31140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:38:11.145673 31140 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:11.146262 31140 leader_election.cc:302] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a09f31796645d790947613b2a9df85; no voters:
I20260929 16:38:11.146863 31140 leader_election.cc:288] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:38:11.147003 31144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:38:11.147982 31144 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [term 2 LEADER]: Becoming Leader. State: Replica: a7a09f31796645d790947613b2a9df85, State: Running, Role: LEADER
I20260929 16:38:11.148499 31144 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } }
I20260929 16:38:11.148938 31140 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:11.151057 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a09f31796645d790947613b2a9df85. Latest consensus state: current_term: 2 leader_uuid: "a7a09f31796645d790947613b2a9df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } } }
I20260929 16:38:11.151077 31145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a7a09f31796645d790947613b2a9df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a09f31796645d790947613b2a9df85" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 35785 } } }
I20260929 16:38:11.151467 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:11.151465 31145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:11.160375 31151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:11.165767 31151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:11.167203 31151 catalog_manager.cc:1259] Loaded cluster ID: a2d5560237c9463c941a06cdcf576388
I20260929 16:38:11.167366 31151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:11.171329 31151 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:11.172780 31151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Loaded TSK: 0
I20260929 16:38:11.174402 31151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:38:11.176937 31140 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 16:38:11.537832 31142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:11.538398 31142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:11.538604 31142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:11.548849 31142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:11.549088 31142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:11.549345 31142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:11.563582 31142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:40981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=35831
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:11.565434 31142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:11.567327 31142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:11.588145 31177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:11.591952 31178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:11.606804 31142 server_base.cc:1034] running on GCE node
W20260929 16:38:11.607131 31180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:11.607901 31142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:11.609015 31142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:11.610376 31142 hybrid_clock.cc:648] HybridClock initialized: now 1790699891610313 us; error 64 us; skew 500 ppm
I20260929 16:38:11.610742 31142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:11 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699891, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:11.614821 31142 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:11.616686 31142 webserver.cc:460] Webserver started at http://127.17.205.1:35831/ using document root <none> and password file <none>
I20260929 16:38:11.617431 31142 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:11.617568 31142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:38:11 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699891, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:11.661916 31142 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.004s sys 0.012s
I20260929 16:38:11.678001 31188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:11.680301 31142 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.004s
I20260929 16:38:11.680639 31142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "a8772db5bd2a483696537a0b2e2c32d3"
format_stamp: "Formatted at 2026-09-29 16:38:08 on dist-test-slave-3h01"
server_key: "ed103c3ce2725508524226ed8cf7b3b0"
server_key_iv: "2c7803e9b206cf7364af42775e1e7a8c"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:11.682235 31142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:11.728025 31142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:11.733706 31142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:11.734117 31142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:11.742975 31142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:11.747777 31142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:11.747926 31142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:11.748095 31142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:11.748168 31142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:11.863008 31142 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:40981
I20260929 16:38:11.863054 31301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:40981 every 8 connection(s)
I20260929 16:38:11.864540 31142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:11.870342 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31142
I20260929 16:38:11.871126 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:45021
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:35785
--webserver_port=36043
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:11 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699891, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:11.908778 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:11.867764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.1:43461)
0929 16:38:11.882953 (+ 15189us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:11.883017 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:11.884877 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:11.887372 (+ 2495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:11.887385 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:11.888054 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:11.888845 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:11.888889 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:11.891412 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:11.891418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:11.892913 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:11.892928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:11.894713 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:11.894723 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:11.895021 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:11.895910 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:11.895943 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:11.898694 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:11.903204 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:11.903243 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:11.903262 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:11.903922 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:11.904487 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:11.904492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:11.904496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:11.904770 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:11.905930 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:11.905939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:11.906386 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 16:38:11.906683 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 16:38:11.907289 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14868,"thread_start_us":14661,"threads_started":1}
I20260929 16:38:11.911504 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:11.867627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43461 (local address 127.17.205.62:35785)
0929 16:38:11.885608 (+ 17981us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:11.885628 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:11.885679 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:38:11.886165 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:11.886186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:11.886756 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:11.887142 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:11.889138 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:11.891119 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:11.893076 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:11.894088 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:11.894294 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:11.899038 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:11.899050 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:11.899072 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:11.899124 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:11.903004 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:11.904110 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:11.904120 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:11.904129 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:11.904330 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:11.904910 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:11.904916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:11.904919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:11.905638 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:11.905843 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:11.909627 (+ 3784us) server_negotiation.cc:299] Negotiation successful
0929 16:38:11.910113 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17717,"thread_start_us":19485,"threads_started":1}
I20260929 16:38:11.915885 31302 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:11.916205 31302 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:11.916954 31302 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
W20260929 16:38:11.920261 31302 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:35785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:38:11.920512 31302 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:11.920823 31302 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:11.922260 31302 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:11.922660 31302 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
W20260929 16:38:11.923899 31302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:38:12.275030 31306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:12.275403 31306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:12.275555 31306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:12.283016 31306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:12.283152 31306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:12.283368 31306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:12.293819 31306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:45021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=36043
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:12.295614 31306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:12.297416 31306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:12.324483 31320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:12.324680 31321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:12.328617 31323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:12.329548 31306 server_base.cc:1034] running on GCE node
I20260929 16:38:12.330508 31306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:12.336325 31306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:12.337654 31306 hybrid_clock.cc:648] HybridClock initialized: now 1790699892337562 us; error 75 us; skew 500 ppm
I20260929 16:38:12.338111 31306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:12 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:12.342350 31306 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:12.344130 31306 webserver.cc:460] Webserver started at http://127.17.205.2:36043/ using document root <none> and password file <none>
I20260929 16:38:12.345033 31306 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:12.345197 31306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:38:12 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:12.390035 31306 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:38:12.393859 31333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:12.395761 31306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:38:12.395951 31306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "1936341fc41343e0bd4500dc0bcfe318"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "414f4842fcc58282e9e36fdf97d0a0fe"
server_key_iv: "d1e8a3e492d58ee8cbb1207045276887"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:12.397449 31306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 16:38:12 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:12.422307 31306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:12.427856 31306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:12.428259 31306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:12.429595 31306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:12.437703 31306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:12.437898 31306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 16:38:12.438060 31306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:12.438259 31306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:12.521842 31306 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:45021
I20260929 16:38:12.521924 31446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:45021 every 8 connection(s)
I20260929 16:38:12.523816 31306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:12.524847 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31306
I20260929 16:38:12.525620 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.205.20:59149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:46581
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:35785
--webserver_port=44971
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:12 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:12.566171 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:12.528545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.2:51295)
0929 16:38:12.537042 (+ 8497us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:12.537110 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:12.541412 (+ 4302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:12.542658 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:12.542671 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:12.543471 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:12.544475 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:12.544507 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:12.546146 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:12.546152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:12.548289 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:12.548306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:12.549339 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:12.549348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:12.549646 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:12.550462 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:12.550495 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:12.553296 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:12.560919 (+ 7623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:12.560931 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:12.560953 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:12.561859 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:12.562316 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:12.562322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:12.562326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:12.562622 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:12.563360 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:12.563370 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:12.563846 (+ 476us) client_negotiation.cc:769] Sending connection context
0929 16:38:12.564142 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 16:38:12.564746 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8089,"thread_start_us":7800,"threads_started":1}
I20260929 16:38:12.567562 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:12.529210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:51295 (local address 127.17.205.62:35785)
0929 16:38:12.538876 (+ 9666us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:12.538881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:12.538906 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:38:12.541859 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:12.541866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:12.541967 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:12.542108 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:12.544740 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:12.545899 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:12.548461 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:12.549079 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:12.549211 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:12.553596 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:12.553604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:12.553608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:12.553655 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:12.560707 (+ 7052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:12.562017 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:12.562024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:12.562028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:12.562190 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:12.562764 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:12.562769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:12.562773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:12.563124 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:12.563288 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:12.566919 (+ 3631us) server_negotiation.cc:299] Negotiation successful
0929 16:38:12.567169 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9561,"thread_start_us":9433,"threads_started":1}
I20260929 16:38:12.569022 31447 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:12.569370 31447 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:12.570266 31447 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:12.572427 31105 ts_manager.cc:194] Registered new tserver with Master: 1936341fc41343e0bd4500dc0bcfe318 (127.17.205.2:45021)
I20260929 16:38:12.574471 31105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:51295
Sep 29 16:38:12 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:12.924041 31302 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:12.924475 31302 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:12.925841 31105 ts_manager.cc:194] Registered new tserver with Master: a8772db5bd2a483696537a0b2e2c32d3 (127.17.205.1:40981)
I20260929 16:38:12.927124 31105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:43461
W20260929 16:38:12.942767 31451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:12.943051 31451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:12.943203 31451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:12.950575 31451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:12.950702 31451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:12.950889 31451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:12.962399 31451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.205.20:59149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:46581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=44971
--tserver_master_addrs=127.17.205.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:12.964074 31451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:12.965749 31451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:12.983204 31456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:12.987775 31457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:12.990089 31459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:12.996877 31451 server_base.cc:1034] running on GCE node
I20260929 16:38:12.998133 31451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:12.999473 31451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:13.000797 31451 hybrid_clock.cc:648] HybridClock initialized: now 1790699893000729 us; error 56 us; skew 500 ppm
I20260929 16:38:13.001188 31451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:13 dist-test-slave-3h01 krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699893, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:13.005389 31451 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:13.007391 31451 webserver.cc:460] Webserver started at http://127.17.205.3:44971/ using document root <none> and password file <none>
I20260929 16:38:13.008311 31451 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:13.008463 31451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:38:13 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699893, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:13.039553 31451 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:38:13.049441 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:13.051628 31451 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260929 16:38:13.051924 31451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "d8f82a01cef14aa39df03ed3e68aefc1"
format_stamp: "Formatted at 2026-09-29 16:38:09 on dist-test-slave-3h01"
server_key: "6595dec294e2c0f0b1bfbcb674aaad18"
server_key_iv: "e4f1efb4c9e6e51da8902e2c8b0244b1"
server_key_version: "kuduclusterkey@0"
I20260929 16:38:13.053433 31451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:13.091540 31451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:13.097002 31451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:13.097332 31451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:13.098582 31451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:13.100481 31451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:13.100605 31451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:13.100728 31451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:13.100802 31451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:38:13 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699892, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for HTTP/127.17.205.20@KRBTEST.COM
I20260929 16:38:13.220222 31451 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:46581
I20260929 16:38:13.220279 31580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:46581 every 8 connection(s)
I20260929 16:38:13.222126 31451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:13.228883 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31451
I20260929 16:38:13.290911 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.230309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.17.205.62:35785)
0929 16:38:13.268866 (+ 38557us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.268874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.268905 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:38:13.268994 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.269001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.269142 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.269311 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.272204 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.273538 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.277263 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.277939 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.278083 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.282450 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.282458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.282463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.282515 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.286048 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.287330 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.287336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.287341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.287499 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.288026 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.288031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.288034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.288500 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.288763 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.290171 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.290455 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":38399,"thread_start_us":33546,"threads_started":1}
Sep 29 16:38:13 dist-test-slave-3h01 krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699893, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:13.313202 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.226088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:44991 (local address 127.17.205.62:35785)
0929 16:38:13.263560 (+ 37472us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.263568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.276828 (+ 13260us) server_negotiation.cc:407] Connection header received
0929 16:38:13.282992 (+ 6164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.282999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.283120 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.283504 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.286249 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.287500 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.289967 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.290691 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.290790 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.296229 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.296238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.296244 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.296290 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.299953 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.303681 (+ 3728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.303688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.303693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.303875 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.305825 (+ 1950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.305830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.305834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.306209 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.306344 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.312473 (+ 6129us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.312750 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37316,"thread_start_us":45149,"threads_started":1}
I20260929 16:38:13.316022 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.226872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:35785 (local address 127.17.205.3:44991)
0929 16:38:13.276657 (+ 49785us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.276717 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.282616 (+ 5899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.284270 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.284286 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.285180 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:13.286078 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.286106 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.287698 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.287704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.289789 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.289806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.291177 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.291185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.291527 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.292278 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:13.292321 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:13.295941 (+ 3620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:13.300135 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:13.300144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:13.300162 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:13.303450 (+ 3288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:13.304013 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:13.304019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:13.304023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:13.305673 (+ 1650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:13.311356 (+ 5683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:13.311367 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:13.311922 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.313785 (+ 1863us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.314432 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49447,"thread_start_us":49208,"threads_started":1}
I20260929 16:38:13.317734 31581 heartbeater.cc:347] Connected to a master server at 127.17.205.62:35785
I20260929 16:38:13.318048 31581 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:13.318908 31581 heartbeater.cc:511] Master 127.17.205.62:35785 requested a full tablet report, sending...
I20260929 16:38:13.320775 31104 ts_manager.cc:194] Registered new tserver with Master: d8f82a01cef14aa39df03ed3e68aefc1 (127.17.205.3:46581)
I20260929 16:38:13.322218 31104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:44991
I20260929 16:38:13.333511 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:38:13.369354 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.350475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.17.205.62:35785)
0929 16:38:13.350666 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.350673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.353322 (+ 2649us) server_negotiation.cc:407] Connection header received
0929 16:38:13.353503 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.353507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.353588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.353674 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.355682 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.356781 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.359778 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.360420 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.360531 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.363489 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.363494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.363498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.363535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.366022 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.366874 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.366879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.366883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.367036 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.367541 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.367547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.367550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.367946 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.368172 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.368774 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.369016 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:38:13.375562 31104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:38:13.383085 31104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:38:13.436297 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.399400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40981 (local address 127.0.0.1:54230)
0929 16:38:13.425694 (+ 26294us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.425766 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.426043 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.427500 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.427509 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.427545 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:13.427986 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.428003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.430123 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.430129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.432762 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.432781 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.435278 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.435286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.435499 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.435556 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.435690 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.435919 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25949,"thread_start_us":27058,"threads_started":1}
I20260929 16:38:13.437238 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.399982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.17.205.1:40981)
0929 16:38:13.423111 (+ 23129us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.423124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.426406 (+ 3282us) server_negotiation.cc:407] Connection header received
0929 16:38:13.426560 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.426570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.426881 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.427157 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:13.428227 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.429919 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.432980 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.434580 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.434717 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.435012 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.436716 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.436909 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22889,"thread_start_us":23172,"threads_started":1}
I20260929 16:38:13.442296 31237 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce (DEFAULT_TABLE table=test-table [id=e2464a69a9d64d71809e448b9463fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:13.444188 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab485e82e9446c8a39b32ffdab06dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:13.448072 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.399625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:46581 (local address 127.0.0.1:34554)
0929 16:38:13.426700 (+ 27075us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.426717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.426824 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.440753 (+ 13929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.440759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.440790 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:13.441162 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.441175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.443351 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.443356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.445350 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.445365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.447237 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.447244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.447454 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.447490 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.447562 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.447706 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26958,"thread_start_us":28556,"threads_started":1}
I20260929 16:38:13.449221 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.400013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.17.205.3:46581)
0929 16:38:13.440111 (+ 40098us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.440120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.440156 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:38:13.440271 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.440279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.440535 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.441405 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:13.441687 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.443162 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.445550 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.446949 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.448339 (+ 1390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.448575 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.448653 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.448883 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39683,"thread_start_us":39795,"threads_started":1}
I20260929 16:38:13.465682 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce (DEFAULT_TABLE table=test-table [id=e2464a69a9d64d71809e448b9463fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:13.468938 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab485e82e9446c8a39b32ffdab06dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:13.470043 31605 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Bootstrap starting.
I20260929 16:38:13.475291 31605 tablet_bootstrap.cc:654] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:13.476430 31601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.428927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.17.205.2:45021)
0929 16:38:13.450279 (+ 21352us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.450289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.450320 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:38:13.450430 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.450438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.450710 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.450922 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:13.457445 (+ 6523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.459111 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.461840 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.468822 (+ 6982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.469024 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.469280 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.475819 (+ 6539us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.476055 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21164,"thread_start_us":22317,"threads_started":1}
I20260929 16:38:13.476815 31605 log.cc:824] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:13.477531 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.428653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:45021 (local address 127.0.0.1:35698)
0929 16:38:13.438129 (+ 9476us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.438148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.438274 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.456879 (+ 18605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.456886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.456932 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:13.457168 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.457187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.459363 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.459369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.461596 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.461617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.475152 (+ 13535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.475162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.475491 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.475539 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.476991 (+ 1452us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.477184 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9360,"thread_start_us":30985,"threads_started":1}
I20260929 16:38:13.481006 31605 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: No bootstrap required, opened a new log
I20260929 16:38:13.481410 31605 ts_tablet_manager.cc:1397] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 16:38:13.482281 31381 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce (DEFAULT_TABLE table=test-table [id=e2464a69a9d64d71809e448b9463fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:13.484086 31381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab485e82e9446c8a39b32ffdab06dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:13.492903 31605 raft_consensus.cc:348] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.493590 31605 raft_consensus.cc:374] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:13.493765 31605 raft_consensus.cc:729] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8772db5bd2a483696537a0b2e2c32d3, State: Initialized, Role: FOLLOWER
I20260929 16:38:13.494581 31605 consensus_queue.cc:260] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.502034 31302 heartbeater.cc:503] Master 127.17.205.62:35785 was elected leader, sending a full tablet report...
I20260929 16:38:13.503399 31605 ts_tablet_manager.cc:1428] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20260929 16:38:13.504642 31611 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Bootstrap starting.
I20260929 16:38:13.508908 31611 tablet_bootstrap.cc:654] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:13.510519 31611 log.cc:824] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:13.511713 31613 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Bootstrap starting.
I20260929 16:38:13.514609 31611 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: No bootstrap required, opened a new log
I20260929 16:38:13.514899 31611 ts_tablet_manager.cc:1397] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260929 16:38:13.516140 31613 tablet_bootstrap.cc:654] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:13.517422 31613 log.cc:824] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:13.522117 31613 tablet_bootstrap.cc:492] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: No bootstrap required, opened a new log
I20260929 16:38:13.522533 31613 ts_tablet_manager.cc:1397] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260929 16:38:13.523711 31611 raft_consensus.cc:348] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.524243 31611 raft_consensus.cc:374] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:13.524358 31611 raft_consensus.cc:729] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f82a01cef14aa39df03ed3e68aefc1, State: Initialized, Role: FOLLOWER
I20260929 16:38:13.525154 31611 consensus_queue.cc:260] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.536324 31613 raft_consensus.cc:348] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.536971 31613 raft_consensus.cc:374] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:13.537159 31613 raft_consensus.cc:729] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1936341fc41343e0bd4500dc0bcfe318, State: Initialized, Role: FOLLOWER
I20260929 16:38:13.538189 31613 consensus_queue.cc:260] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.538753 31581 heartbeater.cc:503] Master 127.17.205.62:35785 was elected leader, sending a full tablet report...
I20260929 16:38:13.539590 31611 ts_tablet_manager.cc:1428] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Time spent starting tablet: real 0.025s user 0.011s sys 0.015s
I20260929 16:38:13.543339 31447 heartbeater.cc:503] Master 127.17.205.62:35785 was elected leader, sending a full tablet report...
I20260929 16:38:13.545038 31613 ts_tablet_manager.cc:1428] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Time spent starting tablet: real 0.022s user 0.010s sys 0.007s
I20260929 16:38:13.548046 31617 raft_consensus.cc:482] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:13.548265 31617 raft_consensus.cc:504] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
W20260929 16:38:13.548676 31448 tablet.cc:2369] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:13.550138 31617 leader_election.cc:288] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8772db5bd2a483696537a0b2e2c32d3 (127.17.205.1:40981), d8f82a01cef14aa39df03ed3e68aefc1 (127.17.205.3:46581)
I20260929 16:38:13.558703 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.550680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:41489 (local address 127.17.205.1:40981)
0929 16:38:13.550895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.550902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.552274 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 16:38:13.552446 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.552451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.552531 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.552616 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:13.553433 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.554566 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.557019 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.557773 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.557929 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.558007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.558260 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.558412 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:38:13.558838 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.550345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40981 (local address 127.17.205.2:41489)
0929 16:38:13.552168 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.552183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.552313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.552784 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.552789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.552821 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:13.553225 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.553235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.554695 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.554700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.556810 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.556823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.557970 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.557976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.558117 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.558141 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.558219 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.558342 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"mutex_wait_us":41,"spinlock_wait_cycles":23296,"thread_start_us":1531,"threads_started":1}
I20260929 16:38:13.559558 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.551538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:46077 (local address 127.17.205.3:46581)
0929 16:38:13.551724 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.551730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.552885 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 16:38:13.553017 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.553023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.553097 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.553190 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:13.553712 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.554390 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.556985 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.558170 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.558330 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.558399 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.559146 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.559317 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:38:13.559932 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab485e82e9446c8a39b32ffdab06dce" candidate_uuid: "1936341fc41343e0bd4500dc0bcfe318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8772db5bd2a483696537a0b2e2c32d3" is_pre_election: true
I20260929 16:38:13.560356 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.551080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:46581 (local address 127.17.205.2:46077)
0929 16:38:13.552787 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:13.552802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:13.552910 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:13.553224 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:13.553229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:13.553252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:13.553564 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.553572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.554516 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.554522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.556808 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:13.556823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.558557 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.558564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:13.558743 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.558789 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:38:13.559852 (+ 1063us) client_negotiation.cc:240] Negotiation successful
0929 16:38:13.560001 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"mutex_wait_us":8,"thread_start_us":1460,"threads_started":1}
I20260929 16:38:13.560537 31257 raft_consensus.cc:2454] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1936341fc41343e0bd4500dc0bcfe318 in term 0.
I20260929 16:38:13.561259 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab485e82e9446c8a39b32ffdab06dce" candidate_uuid: "1936341fc41343e0bd4500dc0bcfe318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" is_pre_election: true
I20260929 16:38:13.561689 31536 raft_consensus.cc:2454] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1936341fc41343e0bd4500dc0bcfe318 in term 0.
I20260929 16:38:13.562336 31336 leader_election.cc:302] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1936341fc41343e0bd4500dc0bcfe318, a8772db5bd2a483696537a0b2e2c32d3; no voters:
I20260929 16:38:13.562999 31617 raft_consensus.cc:2781] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:38:13.563118 31617 raft_consensus.cc:482] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:38:13.563237 31617 raft_consensus.cc:3037] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:13.565351 31617 raft_consensus.cc:504] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.565950 31617 leader_election.cc:288] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [CANDIDATE]: Term 1 election: Requested vote from peers a8772db5bd2a483696537a0b2e2c32d3 (127.17.205.1:40981), d8f82a01cef14aa39df03ed3e68aefc1 (127.17.205.3:46581)
I20260929 16:38:13.566430 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab485e82e9446c8a39b32ffdab06dce" candidate_uuid: "1936341fc41343e0bd4500dc0bcfe318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8772db5bd2a483696537a0b2e2c32d3"
I20260929 16:38:13.566463 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab485e82e9446c8a39b32ffdab06dce" candidate_uuid: "1936341fc41343e0bd4500dc0bcfe318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f82a01cef14aa39df03ed3e68aefc1"
I20260929 16:38:13.566648 31536 raft_consensus.cc:3037] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:13.566642 31257 raft_consensus.cc:3037] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:13.569722 31257 raft_consensus.cc:2454] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1936341fc41343e0bd4500dc0bcfe318 in term 1.
I20260929 16:38:13.569722 31536 raft_consensus.cc:2454] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1936341fc41343e0bd4500dc0bcfe318 in term 1.
I20260929 16:38:13.570125 31336 leader_election.cc:302] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1936341fc41343e0bd4500dc0bcfe318, a8772db5bd2a483696537a0b2e2c32d3; no voters:
I20260929 16:38:13.570701 31617 raft_consensus.cc:2781] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:13.571199 31617 raft_consensus.cc:686] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 1 LEADER]: Becoming Leader. State: Replica: 1936341fc41343e0bd4500dc0bcfe318, State: Running, Role: LEADER
I20260929 16:38:13.571830 31617 consensus_queue.cc:237] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } }
I20260929 16:38:13.577219 31104 catalog_manager.cc:5495] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1936341fc41343e0bd4500dc0bcfe318 (127.17.205.2). New cstate: current_term: 1 leader_uuid: "1936341fc41343e0bd4500dc0bcfe318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1936341fc41343e0bd4500dc0bcfe318" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 45021 } health_report { overall_health: HEALTHY } } }
I20260929 16:38:13.607628 31601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.602211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.17.205.2:45021)
0929 16:38:13.602396 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.602402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.602422 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:38:13.602574 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.602578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.602653 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.602803 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:38:13.603309 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.604111 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.605450 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.605853 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.605969 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.606471 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:38:13.607026 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:38:13.607165 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.607277 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.607366 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:38:13.615970 31536 raft_consensus.cc:1261] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 1 FOLLOWER]: Refusing update from remote peer 1936341fc41343e0bd4500dc0bcfe318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:13.616053 31257 raft_consensus.cc:1261] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 1 FOLLOWER]: Refusing update from remote peer 1936341fc41343e0bd4500dc0bcfe318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 16:38:13.617208 31303 tablet.cc:2369] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:13.617055 31620 consensus_queue.cc:1035] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f82a01cef14aa39df03ed3e68aefc1" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:13.617483 31617 consensus_queue.cc:1035] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8772db5bd2a483696537a0b2e2c32d3" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:13.656296 31105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:13.656567 31105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:13.660331 31105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a09f31796645d790947613b2a9df85: Sending DeleteTablet for 3 replicas of tablet 7ab485e82e9446c8a39b32ffdab06dce
I20260929 16:38:13.661496 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:54230
I20260929 16:38:13.662489 31633 tablet_replica.cc:330] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: stopping tablet replica
I20260929 16:38:13.662550 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:34554
I20260929 16:38:13.662951 31633 raft_consensus.cc:2229] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:13.663513 31633 raft_consensus.cc:2258] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:13.665522 31633 ts_tablet_manager.cc:1905] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:13.667266 31381 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab485e82e9446c8a39b32ffdab06dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:13 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:35698
I20260929 16:38:13.669113 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31142
I20260929 16:38:13.672384 31633 ts_tablet_manager.cc:1918] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:13.672547 31633 log.cc:1196] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/7ab485e82e9446c8a39b32ffdab06dce
I20260929 16:38:13.673072 31633 ts_tablet_manager.cc:1939] T 7ab485e82e9446c8a39b32ffdab06dce P a8772db5bd2a483696537a0b2e2c32d3: Deleting consensus metadata
I20260929 16:38:13.673840 31635 tablet_replica.cc:330] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: stopping tablet replica
I20260929 16:38:13.674348 31635 raft_consensus.cc:2229] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:38:13.675046 31635 raft_consensus.cc:2258] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:13.677063 31091 catalog_manager.cc:4841] TS a8772db5bd2a483696537a0b2e2c32d3 (127.17.205.1:40981): tablet 7ab485e82e9446c8a39b32ffdab06dce (table test-table [id=e2464a69a9d64d71809e448b9463fe4e]) successfully deleted
I20260929 16:38:13.679132 31635 ts_tablet_manager.cc:1905] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:13.686929 31635 ts_tablet_manager.cc:1918] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:13.687428 31635 log.cc:1196] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/7ab485e82e9446c8a39b32ffdab06dce
I20260929 16:38:13.688146 31635 ts_tablet_manager.cc:1939] T 7ab485e82e9446c8a39b32ffdab06dce P 1936341fc41343e0bd4500dc0bcfe318: Deleting consensus metadata
I20260929 16:38:13.690121 31092 catalog_manager.cc:4841] TS 1936341fc41343e0bd4500dc0bcfe318 (127.17.205.2:45021): tablet 7ab485e82e9446c8a39b32ffdab06dce (table test-table [id=e2464a69a9d64d71809e448b9463fe4e]) successfully deleted
I20260929 16:38:13.695422 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.671050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.17.205.1:40981)
0929 16:38:13.671280 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.671286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.671305 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:38:13.671358 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.671363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.671446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.671535 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.675299 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.676391 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.678698 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.679312 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.679442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.688290 (+ 8848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.688308 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.688329 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.688395 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.692192 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.693088 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.693098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.693102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.693243 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.693667 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.693672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.693676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.694062 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.694343 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.694677 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.694884 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:38:13.696115 31297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:13.700138 31634 tablet_replica.cc:330] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: stopping tablet replica
I20260929 16:38:13.700806 31634 raft_consensus.cc:2229] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:13.701508 31634 raft_consensus.cc:2258] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:13.704432 31634 ts_tablet_manager.cc:1905] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:13.714970 31634 ts_tablet_manager.cc:1918] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:13.715252 31634 log.cc:1196] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/7ab485e82e9446c8a39b32ffdab06dce
I20260929 16:38:13.715848 31634 ts_tablet_manager.cc:1939] T 7ab485e82e9446c8a39b32ffdab06dce P d8f82a01cef14aa39df03ed3e68aefc1: Deleting consensus metadata
I20260929 16:38:13.717896 31092 catalog_manager.cc:4841] TS d8f82a01cef14aa39df03ed3e68aefc1 (127.17.205.3:46581): tablet 7ab485e82e9446c8a39b32ffdab06dce (table test-table [id=e2464a69a9d64d71809e448b9463fe4e]) successfully deleted
I20260929 16:38:13.807766 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31142
I20260929 16:38:13.823016 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31306
I20260929 16:38:13.836921 31601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.823805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.17.205.2:45021)
0929 16:38:13.824328 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.824333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.824349 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:38:13.824401 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.824405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.824475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.824565 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.825824 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.826745 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.828106 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.828528 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.828647 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.831072 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.831088 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.831105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.831141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.834051 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.834751 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.834759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.834762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.834928 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.835325 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.835331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.835334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.835734 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.836060 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.836328 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.836546 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417}
I20260929 16:38:13.837693 31442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:13.946732 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31306
I20260929 16:38:13.961794 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31451
I20260929 16:38:13.974303 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:13.962511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.17.205.3:46581)
0929 16:38:13.962705 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:13.962710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:13.962728 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:38:13.962909 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:13.962914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:13.962980 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:13.963069 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:13.964357 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.965081 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.966289 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:13.966660 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:13.966749 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:13.969149 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:13.969165 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:13.969181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:13.969213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:13.971867 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.972542 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.972549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.972551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.972672 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:13.972959 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:13.972962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:13.972964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:13.973297 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:13.973522 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:13.973829 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:38:13.973997 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:38:13.975023 31576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:14.062738 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31451
I20260929 16:38:14.075680 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31072
I20260929 16:38:14.076859 31135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:14.188356 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31072
I20260929 16:38:14.209324 18228 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:38:14.543659 18228 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:38:14.543850 18228 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:38:14.879105 18228 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:38:14.879 UTC [30124] LOG: received smart shutdown request
2026-09-29 16:38:14.882 UTC [30124] LOG: background worker "logical replication launcher" (PID 30133) exited with exit code 1
2026-09-29 16:38:14.882 UTC [30128] LOG: shutting down
2026-09-29 16:38:15.019 UTC [30124] LOG: database system is shut down
2026-09-29T16:38:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50539 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:38:15 dist-test-slave-3h01 krb5kdc[31648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:38:15 dist-test-slave-3h01 krb5kdc[31648](info): set up 2 sockets
Sep 29 16:38:15 dist-test-slave-3h01 krb5kdc[31648](info): commencing operation
krb5kdc: starting...
W20260929 16:38:16.127367 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:38:16 dist-test-slave-3h01 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:38:16.351158 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34433
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:16.678802 31661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:16.679104 31661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:16.679246 31661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:16.686625 31661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:16.686738 31661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:16.686810 31661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:16.686892 31661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:16.686944 31661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:16.696934 31661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:34433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:16.698312 31661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:16.699652 31661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:16.711021 31666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:16.711082 31667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:16.711051 31669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:16.712049 31661 server_base.cc:1034] running on GCE node
I20260929 16:38:16.712796 31661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:16.714180 31661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:16.715369 31661 hybrid_clock.cc:648] HybridClock initialized: now 1790699896715328 us; error 32 us; skew 500 ppm
I20260929 16:38:16.715698 31661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:16 dist-test-slave-3h01 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:16.718596 31661 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:16.720096 31661 webserver.cc:460] Webserver started at http://127.17.205.62:42119/ using document root <none> and password file <none>
I20260929 16:38:16.720774 31661 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:16.720888 31661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:16.721165 31661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:16.724251 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f"
format_stamp: "Formatted at 2026-09-29 16:38:16 on dist-test-slave-3h01"
server_key: "cc3e65b387a63d87d08824c205b18a1e"
server_key_iv: "5f06a519b3539ffd996ad678031667c5"
server_key_version: "encryptionkey@0"
I20260929 16:38:16.725039 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f"
format_stamp: "Formatted at 2026-09-29 16:38:16 on dist-test-slave-3h01"
server_key: "cc3e65b387a63d87d08824c205b18a1e"
server_key_iv: "5f06a519b3539ffd996ad678031667c5"
server_key_version: "encryptionkey@0"
I20260929 16:38:16.730047 31661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:38:16.733536 31676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:16.734848 31661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:38:16.735006 31661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f"
format_stamp: "Formatted at 2026-09-29 16:38:16 on dist-test-slave-3h01"
server_key: "cc3e65b387a63d87d08824c205b18a1e"
server_key_iv: "5f06a519b3539ffd996ad678031667c5"
server_key_version: "encryptionkey@0"
I20260929 16:38:16.735186 31661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:16.756994 31661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:16.761138 31661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:16.761425 31661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:16.778877 31661 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:34433
I20260929 16:38:16.778896 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:34433 every 8 connection(s)
I20260929 16:38:16.780252 31661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:16.783064 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31661
I20260929 16:38:16.783305 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:38:16.783912 18228 external_mini_cluster.cc:1428] Setting key e6144f99ad8c17adfaa20ee82f9ba034
I20260929 16:38:16.784103 31728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:16.790588 31728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.205.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.61@KRBTEST.COM" created.
Sep 29 16:38:16 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:16.813186 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.793818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.17.205.62:34433)
0929 16:38:16.794628 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:16.794643 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:16.795113 (+ 470us) server_negotiation.cc:407] Connection header received
0929 16:38:16.797100 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:16.797105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:16.797343 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:16.797528 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:16.798794 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:16.800043 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:16.801392 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:16.802044 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:16.802187 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:16.805643 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:16.805651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:16.805664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:16.805697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:16.808237 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:16.809051 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:16.809061 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:16.809065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:16.809263 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:16.809830 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:16.809834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:16.809838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:16.810541 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:16.810752 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:16.811456 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 16:38:16.812039 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":573,"thread_start_us":371,"threads_started":1}
I20260929 16:38:16.813241 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.793839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34433 (local address 127.0.0.1:35578)
0929 16:38:16.794949 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.794983 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:16.796735 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:16.797556 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:16.797562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:16.798270 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:16.798610 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:16.798628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:16.800181 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:16.800185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:16.801249 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:16.801261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:16.802255 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:16.802261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:16.802444 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:16.803002 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:16.803025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:16.805419 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:16.808367 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:16.808379 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:16.808391 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:16.808891 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:16.809383 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:16.809388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:16.809392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:16.809712 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:16.810704 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:16.810712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:16.810952 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 16:38:16.811282 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 16:38:16.812166 (+ 884us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":322,"thread_start_us":717,"threads_started":1}
I20260929 16:38:16.816190 31728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:38:16.817348 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.816911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37308)
0929 16:38:16.817092 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.817205 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:38:16.817648 31680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
I20260929 16:38:16.819185 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.818805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37318)
0929 16:38:16.818956 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.819033 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:38:16.819404 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:37053: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
I20260929 16:38:16.830168 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:37053
--webserver_interface=127.17.205.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:16.868765 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } attempt: 1
I20260929 16:38:16.870081 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.869502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37324)
0929 16:38:16.869726 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.869891 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:38:16.871084 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.870688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37336)
0929 16:38:16.870846 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.870932 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:38:16.871309 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:37053: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
I20260929 16:38:16.939574 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } attempt: 2
I20260929 16:38:16.940840 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.940266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37348)
0929 16:38:16.940549 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.940662 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:38:16.941856 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:16.941457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37356)
0929 16:38:16.941602 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:16.941697 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:38:16.942061 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:37053: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
I20260929 16:38:17.033327 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } attempt: 3
I20260929 16:38:17.034607 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.034024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37364)
0929 16:38:17.034265 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.034405 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:38:17.035753 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.035321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37368)
0929 16:38:17.035473 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.035572 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:38:17.036012 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:37053: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
W20260929 16:38:17.155313 31735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:17.155634 31735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:17.155781 31735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:17.163172 31735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:17.163340 31735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:17.163422 31735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:17.163497 31735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:17.163559 31735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:17.173599 31735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.205.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:17.174990 31735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:17.176383 31735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:38:17.179303 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } attempt: 4
I20260929 16:38:17.180539 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.179917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37374)
0929 16:38:17.180244 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.180368 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 16:38:17.181716 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.181271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37384)
0929 16:38:17.181428 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.181545 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:38:17.181963 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:37053: Network error: Client connection negotiation failed: client connection to 127.17.205.61:37053: connect: Connection refused (error 111)
W20260929 16:38:17.188398 31741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:17.188478 31743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:17.188498 31740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:17.189438 31735 server_base.cc:1034] running on GCE node
I20260929 16:38:17.190219 31735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:17.191777 31735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:17.192983 31735 hybrid_clock.cc:648] HybridClock initialized: now 1790699897192927 us; error 61 us; skew 500 ppm
I20260929 16:38:17.193323 31735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:17.196467 31735 init.cc:377] Logged in from keytab as kudu/127.17.205.61@KRBTEST.COM (short username kudu)
I20260929 16:38:17.197974 31735 webserver.cc:460] Webserver started at http://127.17.205.61:39955/ using document root <none> and password file <none>
I20260929 16:38:17.198652 31735 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:17.198750 31735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:17.199020 31735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:17.202129 31735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data/instance:
uuid: "724a4d1478fa426bba0e14c6c038a0a2"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1c2fb365e0a5bbe50f2d3c772879dcf3"
server_key_iv: "533b2ef5f94b2c72174d8fafc09bd8f1"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.202946 31735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal/instance:
uuid: "724a4d1478fa426bba0e14c6c038a0a2"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1c2fb365e0a5bbe50f2d3c772879dcf3"
server_key_iv: "533b2ef5f94b2c72174d8fafc09bd8f1"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.208277 31735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:38:17.211783 31750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:17.213133 31735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:38:17.213281 31735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "724a4d1478fa426bba0e14c6c038a0a2"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1c2fb365e0a5bbe50f2d3c772879dcf3"
server_key_iv: "533b2ef5f94b2c72174d8fafc09bd8f1"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.213449 31735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:17.225700 31735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:17.229961 31735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:17.230233 31735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:17.249225 31735 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.61:37053
I20260929 16:38:17.249266 31801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.61:37053 every 8 connection(s)
I20260929 16:38:17.250730 31735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
I20260929 16:38:17.254551 31802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:17.260995 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31735
I20260929 16:38:17.261205 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-1/wal/instance
I20260929 16:38:17.261762 18228 external_mini_cluster.cc:1428] Setting key 3605994fca8f91cf2507165d0253f6d9
I20260929 16:38:17.261325 31802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.60@KRBTEST.COM; defaulting to no policy
I20260929 16:38:17.283245 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.264680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.17.205.62:34433)
0929 16:38:17.264889 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.264896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.266267 (+ 1371us) server_negotiation.cc:407] Connection header received
0929 16:38:17.268272 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.268279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.268374 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.268488 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.270063 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.271037 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.272620 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.273250 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.273377 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.277095 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.277101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.277107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.277155 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.279982 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.280966 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.280971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.280973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.281100 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.281577 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.281579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.281581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.281863 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.282001 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.282766 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.282984 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.60@KRBTEST.COM" created.
I20260929 16:38:17.284637 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.265215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34433 (local address 127.0.0.1:35582)
0929 16:38:17.266076 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.266123 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.267975 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.268696 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.268704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.269262 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.269847 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.269865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.271176 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.271180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.272477 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.272491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.273440 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.273445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.273650 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.274300 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.274323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.276800 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.280153 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.280163 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.280184 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.280834 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.281247 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.281251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.281254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.281478 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.281995 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.282002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.282380 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.282657 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.283156 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":362,"threads_started":1}
I20260929 16:38:17.286463 31802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:38:17.300558 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.287278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37386)
0929 16:38:17.287482 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.287502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.287615 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.288327 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.288332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.288776 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.288952 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.288963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.290262 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.290267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.291108 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.291119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.291964 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.291970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.292179 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.292756 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.292778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.295011 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.298040 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.298046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.298050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.298414 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.298761 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.298766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.298770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.298949 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.299613 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.299618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.299757 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.299875 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.300177 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:38:17.301000 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.287354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.17.205.61:37053)
0929 16:38:17.287907 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.287913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.287941 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:38:17.287986 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.287992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.288180 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.288291 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.289096 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.290124 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.291262 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.291817 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.291910 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.295199 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.295237 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.295257 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.295296 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.297908 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.298538 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.298546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.298548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.298662 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.299051 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.299056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.299060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.299479 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.299674 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.300147 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.300566 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":264,"threads_started":1}
I20260929 16:38:17.303403 31802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:38:17.304661 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.304163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53914)
0929 16:38:17.304337 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.304469 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:38:17.304982 31753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.306782 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.306365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53926)
0929 16:38:17.306547 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.306651 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:38:17.307017 31802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
I20260929 16:38:17.310567 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:43163
--webserver_interface=127.17.205.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:17.356364 31802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 1
I20260929 16:38:17.357573 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.357123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53930)
0929 16:38:17.357312 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.357417 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:38:17.358686 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.358336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53942)
0929 16:38:17.358476 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.358567 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:38:17.358914 31802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.427139 31802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 2
I20260929 16:38:17.428301 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.427858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53956)
0929 16:38:17.428057 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.428149 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:38:17.429287 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.428975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53970)
0929 16:38:17.429096 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.429168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:38:17.429514 31802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.481276 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } attempt: 5
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:38:17.495594 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.482132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.17.205.61:37053)
0929 16:38:17.482303 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.482307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.482326 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:38:17.482553 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.482558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.482638 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.482734 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.483791 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.484489 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.486236 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.486693 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.486788 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.490439 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.490445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.490448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.490485 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.492942 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.493778 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.493783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.493785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.493914 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.494309 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.494312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.494314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.494591 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.494701 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.495128 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.495361 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:38:17.495738 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.481982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37390)
0929 16:38:17.482217 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.482240 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.482390 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.482801 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.482806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.483368 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.483608 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.483622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.484647 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.484654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.486087 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.486104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.486847 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.486853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.487037 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.487690 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.487713 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.490221 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.493101 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.493107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.493111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.493650 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.494034 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.494038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.494042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.494206 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.494703 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.494708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.494883 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.495053 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.495417 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:38:17.496881 31728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:38:17.498020 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.497554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53984)
0929 16:38:17.497743 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.497842 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:38:17.498945 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.498590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:53994)
0929 16:38:17.498726 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.498822 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:38:17.499128 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.520704 31802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 3
I20260929 16:38:17.521780 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.521316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54000)
0929 16:38:17.521528 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.521625 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:38:17.522939 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.522499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54014)
0929 16:38:17.522726 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.522806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260929 16:38:17.523145 31802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.550387 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 1
I20260929 16:38:17.551638 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.551056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54030)
0929 16:38:17.551316 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.551439 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:38:17.552594 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.552241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54032)
0929 16:38:17.552380 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.552461 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:38:17.552810 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.621039 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 2
I20260929 16:38:17.622236 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.621699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54036)
0929 16:38:17.621928 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.622043 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:38:17.623282 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.622847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54040)
0929 16:38:17.623031 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.623115 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:38:17.623482 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
W20260929 16:38:17.639616 31809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:17.639914 31809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:17.640010 31809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:17.647151 31809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:17.647279 31809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:17.647334 31809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:17.647389 31809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:17.647431 31809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:17.657308 31809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:34433,127.17.205.61:37053,127.17.205.60:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.205.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:17.658648 31809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:17.659962 31809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:38:17.667318 31802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 4
I20260929 16:38:17.668596 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.668022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54056)
0929 16:38:17.668235 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.668331 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:38:17.669896 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.669554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54062)
0929 16:38:17.669690 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.669767 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 16:38:17.671265 31802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
W20260929 16:38:17.677269 31814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:17.677208 31815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:17.677495 31817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:17.680316 31809 server_base.cc:1034] running on GCE node
I20260929 16:38:17.681046 31809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:17.682694 31809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:17.683943 31809 hybrid_clock.cc:648] HybridClock initialized: now 1790699897683867 us; error 59 us; skew 500 ppm
I20260929 16:38:17.684372 31809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:17.693598 31809 init.cc:377] Logged in from keytab as kudu/127.17.205.60@KRBTEST.COM (short username kudu)
I20260929 16:38:17.695153 31809 webserver.cc:460] Webserver started at http://127.17.205.60:44119/ using document root <none> and password file <none>
I20260929 16:38:17.695895 31809 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:17.695994 31809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:17.696245 31809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:17.699080 31809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data/instance:
uuid: "fcfb31a8cdfc4df1986e30b01e3d606e"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1a81cba12c11236539a02517c037964d"
server_key_iv: "3d8bb650567f3dd31b0baf0360d23f58"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.699873 31809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal/instance:
uuid: "fcfb31a8cdfc4df1986e30b01e3d606e"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1a81cba12c11236539a02517c037964d"
server_key_iv: "3d8bb650567f3dd31b0baf0360d23f58"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.704793 31809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:38:17.708088 31824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:17.709439 31809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:38:17.709585 31809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "fcfb31a8cdfc4df1986e30b01e3d606e"
format_stamp: "Formatted at 2026-09-29 16:38:17 on dist-test-slave-3h01"
server_key: "1a81cba12c11236539a02517c037964d"
server_key_iv: "3d8bb650567f3dd31b0baf0360d23f58"
server_key_version: "encryptionkey@0"
I20260929 16:38:17.709756 31809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:17.729703 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 3
I20260929 16:38:17.730859 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.730407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54076)
0929 16:38:17.730598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.730691 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:38:17.732087 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.731692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54092)
0929 16:38:17.731884 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.731974 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 16:38:17.732286 31728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:43163: Network error: Client connection negotiation failed: client connection to 127.17.205.60:43163: connect: Connection refused (error 111)
I20260929 16:38:17.734462 31809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:17.739712 31809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:17.740031 31809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:17.756147 31809 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.60:43163
I20260929 16:38:17.756166 31875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.60:43163 every 8 connection(s)
I20260929 16:38:17.757489 31809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
I20260929 16:38:17.760584 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31809
I20260929 16:38:17.760766 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699695913193-18228-0/minicluster-data/master-2/wal/instance
I20260929 16:38:17.761226 18228 external_mini_cluster.cc:1428] Setting key 30abe18b063b094f138a0f3dea1dbc67
I20260929 16:38:17.761261 31876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:17.771029 31876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:17.785820 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.763001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.17.205.60:43163)
0929 16:38:17.763843 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.763859 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.763911 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:38:17.766060 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.766092 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.766627 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.767090 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.768433 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.770359 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.771839 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.772816 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.773007 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.777630 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.777638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.777648 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.777683 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.781080 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.781781 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.781787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.781794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.781960 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.782467 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.782472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.782476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.783258 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.783559 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.783995 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.784449 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":383,"threads_started":1}
I20260929 16:38:17.790367 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.773853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.17.205.62:34433)
0929 16:38:17.774030 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.774035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.774728 (+ 693us) server_negotiation.cc:407] Connection header received
0929 16:38:17.774898 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.774904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.774970 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.775050 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.776208 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.776926 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.778575 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.779590 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.779729 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.783498 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.783504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.783509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.783550 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.787459 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.788178 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.788183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.788186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.788367 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.788730 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.788735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.788737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.789055 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.789226 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.789792 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.790010 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:38:17.790400 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.773882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:34433 (local address 127.0.0.1:35584)
0929 16:38:17.774591 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.774632 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.774779 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.775130 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.775137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.775753 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.776034 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.776046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.777079 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.777085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.778395 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.778410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.779774 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.779781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.779965 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.780605 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.780627 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.783277 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.787595 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.787607 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.787611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.788013 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.788496 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.788499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.788501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.788618 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.789299 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.789306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.789494 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.789651 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.789990 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":394,"threads_started":1}
I20260929 16:38:17.791728 31876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:38:17.804975 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.789125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.17.205.62:34433)
0929 16:38:17.789652 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.789656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.789676 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:38:17.789715 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.789719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.789776 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.789868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.791293 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.792333 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.794069 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.794686 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.794807 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.799334 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.799341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.799346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.799389 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.802154 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.802994 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.802999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.803001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.803154 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.803566 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.803572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.803576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.803968 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.804192 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.804473 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.804653 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260929 16:38:17.805791 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.792525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:37053 (local address 127.0.0.1:37406)
0929 16:38:17.792700 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.792718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.792811 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.793094 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.793099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.793532 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.793705 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.793716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.794967 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.794973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.796240 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.796256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.796978 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.796984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.797178 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.797714 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.797737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.799911 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.803276 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.803281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.803285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.803748 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.804131 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.804135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.804139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.804308 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.804885 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.804890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.805068 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.805220 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.805521 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:38:17.805927 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.792596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.17.205.61:37053)
0929 16:38:17.792789 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.792793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.792809 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:38:17.792911 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.792915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.792972 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.793054 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.793862 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.794836 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.796401 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.796830 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.796922 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.800122 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.800129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.800134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.800177 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.803088 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.803869 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.803873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.803875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.804013 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.804424 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.804428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.804431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.804758 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.804899 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.805343 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.805555 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:38:17.806921 31876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:38:17.822564 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.807806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.17.205.60:43163)
0929 16:38:17.807943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.807948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.807963 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:38:17.808091 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.808096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.808154 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.808224 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.809036 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.810270 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.811765 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.812327 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.812438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.816280 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.816287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.816291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.816326 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.819645 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.820440 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.820445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.820450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.820595 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.820982 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.820987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.820990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.821352 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.821468 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.821956 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.822198 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:38:17.823426 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.807721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54112)
0929 16:38:17.807894 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.807910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.808000 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.808251 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.808255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.808710 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.808880 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.808891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.810419 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.810425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.811625 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.811637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.812613 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.812619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.812837 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.813448 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.813471 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.816083 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.819798 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.819805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.819809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.820247 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.820715 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.820719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.820723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.820864 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.821547 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.821552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.821703 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.822777 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.823065 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:38:17.825018 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.806742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.17.205.61:37053)
0929 16:38:17.806964 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.806970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.806988 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:38:17.807090 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.807094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.807164 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.807304 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.808669 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.809513 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.811293 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.812099 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.812222 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.818285 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.818290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.818294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.818329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.821650 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.822470 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.822475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.822477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.822619 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.823169 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.823175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.823178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.823580 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.823802 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.824472 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.824712 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 16:38:17.829031 31876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: Bootstrap starting.
I20260929 16:38:17.829762 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:38:17.832967 31876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:17.834393 31876 log.cc:824] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:17.838485 31876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: No bootstrap required, opened a new log
I20260929 16:38:17.849573 31876 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:17.850235 31876 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:17.850431 31876 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfb31a8cdfc4df1986e30b01e3d606e, State: Initialized, Role: FOLLOWER
I20260929 16:38:17.851356 31876 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:17.852871 31890 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:17.853284 31890 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:17.854414 31876 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:17.856410 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.835901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.17.205.62:34433)
0929 16:38:17.836086 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.836091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.836733 (+ 642us) server_negotiation.cc:407] Connection header received
0929 16:38:17.836929 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.836936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.837027 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.837130 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.839861 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.840997 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.843493 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.844081 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.844169 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.847808 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.847814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.847819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.847863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.851528 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.853518 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.853524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.853528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.853678 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.854170 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.854175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.854178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.854528 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.855006 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.855600 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.856070 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:38:17.857306 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.835446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.17.205.61:37053)
0929 16:38:17.835808 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.835814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.840131 (+ 4317us) server_negotiation.cc:407] Connection header received
0929 16:38:17.840279 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.840285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.840369 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.840472 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.843022 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.844127 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.846432 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.847108 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.847201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.850490 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.850497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.850502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.850551 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.853833 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.854657 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.854663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.854666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.854834 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.855330 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.855335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.855338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.855707 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.855953 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.856523 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.856869 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260929 16:38:17.861430 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.835415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.17.205.60:43163)
0929 16:38:17.835668 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.835674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.843635 (+ 7961us) server_negotiation.cc:407] Connection header received
0929 16:38:17.844067 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.844077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.844163 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.844234 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.846490 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.847670 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.849809 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.850523 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.850618 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.854089 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.854111 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.854116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.854157 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.857394 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.858976 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.858983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.858988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.859268 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.859830 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.859835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.859839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.860220 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.860474 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.860687 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.860959 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260929 16:38:17.872483 31901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:38:17.873561 31901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:38:17.869781 31809 master_runner.cc:186] Error getting master registration for 127.17.205.62:34433: OK, instance_id { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" instance_seqno: 1790699896773941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:38:17.878571 31809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:38:17.909497 31728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 4
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699896, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
I20260929 16:38:17.923926 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.910079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54130)
0929 16:38:17.910286 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.910308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.910434 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.910765 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.910770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.911148 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.911356 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.911367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.912435 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.912440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.913459 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.913473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.914226 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.914232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.914407 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.915094 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.915119 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.917755 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.921639 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.921643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.921646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.922053 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.922453 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.922457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.922459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.922581 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.923160 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.923163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.923336 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.923447 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.923649 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:38:17.924176 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.910130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.17.205.60:43163)
0929 16:38:17.910310 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.910315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.910373 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:38:17.910541 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.910547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.910608 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.910701 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.911491 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.912309 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.913601 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.914089 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.914198 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.917995 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.918002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.918008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.918047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.921511 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.922160 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.922166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.922170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.922351 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.922677 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.922682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.922685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.923047 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.923205 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.923573 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.923812 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:38:17.928524 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: Bootstrap starting.
I20260929 16:38:17.932328 31728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:17.933420 31728 log.cc:824] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:17.936254 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: No bootstrap required, opened a new log
I20260929 16:38:17.943500 31728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:17.943924 31728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:17.944048 31728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766ff3155cdb4a73aa6d6bdaa0626e3f, State: Initialized, Role: FOLLOWER
I20260929 16:38:17.944787 31728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:17.945820 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:17.946209 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:17.946810 31728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:38:17.956086 31915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:38:17.956290 31915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:38:17.955442 31661 master_runner.cc:186] Error getting master registration for 127.17.205.61:37053: OK, instance_id { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" instance_seqno: 1790699897243733 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:38:17.964111 31661 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:38:17.970512 31802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } attempt: 5
Sep 29 16:38:17 dist-test-slave-3h01 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699897, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
I20260929 16:38:17.984375 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.971107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:43163 (local address 127.0.0.1:54134)
0929 16:38:17.971403 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:17.971423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:17.971549 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:17.971858 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:17.971863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:17.972279 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:17.972451 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.972460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.973490 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.973495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.974367 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:17.974379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.975540 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.975547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:17.975806 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.976660 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:17.976684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:17.978935 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:17.981912 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.981917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.981922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.982303 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.982669 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:17.982674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:17.982678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:17.982868 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:17.983400 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:17.983405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:17.983553 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 16:38:17.983677 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:38:17.983969 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 16:38:17.984968 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:17.971160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.17.205.60:43163)
0929 16:38:17.971423 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:17.971428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:17.971477 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:38:17.971664 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:17.971668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:17.971729 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:17.971808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:17.972599 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.973331 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.974531 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:17.975158 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:17.975345 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:17.979110 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:17.979115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:17.979118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:17.979151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:17.981779 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.982420 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.982424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.982426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.982564 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:17.982966 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:17.982970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:17.982972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:17.983293 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:17.983425 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:17.983786 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:38:17.984392 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 16:38:17.989123 31802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: Bootstrap starting.
W20260929 16:38:17.992077 31830 tablet.cc:2369] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:17.997260 31802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:17.999722 31802 log.cc:824] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:18.006129 31802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: No bootstrap required, opened a new log
W20260929 16:38:18.014443 31682 tablet.cc:2369] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:18.018165 31802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:18.018707 31802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:18.018863 31802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724a4d1478fa426bba0e14c6c038a0a2, State: Initialized, Role: FOLLOWER
I20260929 16:38:18.019753 31802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:18.020836 31918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.021224 31918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:18.022367 31802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:18.037998 31735 master_runner.cc:263] No leader master found from master 724a4d1478fa426bba0e14c6c038a0a2
I20260929 16:38:18.038175 31735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:38:18.041489 31918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:18.041767 31918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:18.043416 31918 leader_election.cc:288] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 766ff3155cdb4a73aa6d6bdaa0626e3f (127.17.205.62:34433), fcfb31a8cdfc4df1986e30b01e3d606e (127.17.205.60:43163)
I20260929 16:38:18.044445 31703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724a4d1478fa426bba0e14c6c038a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" is_pre_election: true
I20260929 16:38:18.044498 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724a4d1478fa426bba0e14c6c038a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" is_pre_election: true
I20260929 16:38:18.045071 31703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724a4d1478fa426bba0e14c6c038a0a2 in term 0.
I20260929 16:38:18.045089 31851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724a4d1478fa426bba0e14c6c038a0a2 in term 0.
I20260929 16:38:18.045671 31753 leader_election.cc:302] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724a4d1478fa426bba0e14c6c038a0a2, fcfb31a8cdfc4df1986e30b01e3d606e; no voters:
I20260929 16:38:18.046370 31918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:38:18.046530 31918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 16:38:18.046379 31929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 16:38:18.046629 31918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:18.050302 31918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:18.051641 31703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724a4d1478fa426bba0e14c6c038a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f"
I20260929 16:38:18.051877 31703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:18.055481 31703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724a4d1478fa426bba0e14c6c038a0a2 in term 1.
I20260929 16:38:18.056102 31752 leader_election.cc:302] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724a4d1478fa426bba0e14c6c038a0a2, 766ff3155cdb4a73aa6d6bdaa0626e3f; no voters:
I20260929 16:38:18.064604 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724a4d1478fa426bba0e14c6c038a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e"
I20260929 16:38:18.064877 31851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:18.068408 31851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724a4d1478fa426bba0e14c6c038a0a2 in term 1.
I20260929 16:38:18.069362 31918 leader_election.cc:288] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [CANDIDATE]: Term 1 election: Requested vote from peers 766ff3155cdb4a73aa6d6bdaa0626e3f (127.17.205.62:34433), fcfb31a8cdfc4df1986e30b01e3d606e (127.17.205.60:43163)
I20260929 16:38:18.069852 31918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:18.070304 31918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 724a4d1478fa426bba0e14c6c038a0a2, State: Running, Role: LEADER
W20260929 16:38:18.071314 31929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:38:18.071147 31918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } }
I20260929 16:38:18.091668 31931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724a4d1478fa426bba0e14c6c038a0a2. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.092074 31931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:18.099032 31933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:18.101348 31933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:18.110877 31851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [term 1 FOLLOWER]: Refusing update from remote peer 724a4d1478fa426bba0e14c6c038a0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:18.111904 31931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:18.119444 31703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [term 1 FOLLOWER]: Refusing update from remote peer 724a4d1478fa426bba0e14c6c038a0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:18.120354 31931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:18.140941 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724a4d1478fa426bba0e14c6c038a0a2. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.141242 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:18.150936 31936 mvcc.cc:205] Tried to move back new op lower bound from 7334706782651789312 to 7334706782515240960. Current Snapshot: MvccSnapshot[applied={T|T < 7334706782651789312}]
I20260929 16:38:18.154867 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.155119 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:18.164561 31890 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724a4d1478fa426bba0e14c6c038a0a2. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.164886 31890 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:18.165411 31931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.165702 31931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:18.166699 31933 catalog_manager.cc:1347] Generated new cluster ID: e92dc7cb46114817821233d12acaddcc
I20260929 16:38:18.166795 31933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:18.184446 31918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.184844 31918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:18.205889 31890 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724a4d1478fa426bba0e14c6c038a0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766ff3155cdb4a73aa6d6bdaa0626e3f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 34433 } } peers { permanent_uuid: "724a4d1478fa426bba0e14c6c038a0a2" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 37053 } } peers { permanent_uuid: "fcfb31a8cdfc4df1986e30b01e3d606e" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 43163 } } }
I20260929 16:38:18.206238 31890 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:38:18.221328 31933 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:38:18.223563 31933 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:18.254235 31933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724a4d1478fa426bba0e14c6c038a0a2: Generated new TSK 0
I20260929 16:38:18.255571 31933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:38:18.875669 31901 catalog_manager.cc:1259] Loaded cluster ID: e92dc7cb46114817821233d12acaddcc
I20260929 16:38:18.875789 31901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: loading cluster ID for follower catalog manager: success
I20260929 16:38:18.878250 31901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: acquiring CA information for follower catalog manager: success
I20260929 16:38:18.880137 31901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcfb31a8cdfc4df1986e30b01e3d606e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:38:18.958235 31915 catalog_manager.cc:1259] Loaded cluster ID: e92dc7cb46114817821233d12acaddcc
I20260929 16:38:18.958341 31915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: loading cluster ID for follower catalog manager: success
I20260929 16:38:18.960958 31915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: acquiring CA information for follower catalog manager: success
I20260929 16:38:18.962785 31915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 766ff3155cdb4a73aa6d6bdaa0626e3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:38:19.401631 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31661
I20260929 16:38:19.402911 31723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:19.514323 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31661
I20260929 16:38:19.531603 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31735
I20260929 16:38:19.532618 31797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:19.638556 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31735
I20260929 16:38:19.655386 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31809
I20260929 16:38:19.656513 31871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:19.763461 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31809
2026-09-29T16:38:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4641 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:38:19 dist-test-slave-3h01 krb5kdc[31951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:38:19 dist-test-slave-3h01 krb5kdc[31951](info): set up 2 sockets
Sep 29 16:38:19 dist-test-slave-3h01 krb5kdc[31951](info): commencing operation
krb5kdc: starting...
W20260929 16:38:20.771538 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.009s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:38:20 dist-test-slave-3h01 krb5kdc[31951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:38:20.996898 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:39909
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:39909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:21.321874 31964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:21.322192 31964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:21.322326 31964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:21.329770 31964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:21.329905 31964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:21.329985 31964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:21.330060 31964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:21.330117 31964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:21.340166 31964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:39909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:21.341488 31964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:21.342939 31964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:21.354671 31969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:21.354610 31970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:21.354643 31972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:21.355363 31964 server_base.cc:1034] running on GCE node
I20260929 16:38:21.356117 31964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:21.357546 31964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:21.358744 31964 hybrid_clock.cc:648] HybridClock initialized: now 1790699901358693 us; error 48 us; skew 500 ppm
I20260929 16:38:21.359081 31964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:21 dist-test-slave-3h01 krb5kdc[31951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699901, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:21.362094 31964 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:21.363565 31964 webserver.cc:460] Webserver started at http://127.17.205.62:32827/ using document root <none> and password file <none>
I20260929 16:38:21.364231 31964 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:21.364327 31964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:21.364580 31964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:21.367628 31964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "84050e0865504512ac50e725c49ec24a"
format_stamp: "Formatted at 2026-09-29 16:38:21 on dist-test-slave-3h01"
server_key: "8c16c0209bb05ddca6400cbd4cb00b77"
server_key_iv: "c8c3c7ce695576a63791b7423ea85486"
server_key_version: "encryptionkey@0"
I20260929 16:38:21.368340 31964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "84050e0865504512ac50e725c49ec24a"
format_stamp: "Formatted at 2026-09-29 16:38:21 on dist-test-slave-3h01"
server_key: "8c16c0209bb05ddca6400cbd4cb00b77"
server_key_iv: "c8c3c7ce695576a63791b7423ea85486"
server_key_version: "encryptionkey@0"
I20260929 16:38:21.373399 31964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:38:21.376752 31979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:21.378093 31964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 16:38:21.378232 31964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "84050e0865504512ac50e725c49ec24a"
format_stamp: "Formatted at 2026-09-29 16:38:21 on dist-test-slave-3h01"
server_key: "8c16c0209bb05ddca6400cbd4cb00b77"
server_key_iv: "c8c3c7ce695576a63791b7423ea85486"
server_key_version: "encryptionkey@0"
I20260929 16:38:21.378393 31964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:21.389941 31964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:21.394071 31964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:21.394294 31964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:21.411566 31964 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:39909
I20260929 16:38:21.411587 32030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:39909 every 8 connection(s)
I20260929 16:38:21.412982 31964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:21.417271 32031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:21.418056 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 31964
I20260929 16:38:21.418284 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:38:21.419080 18228 external_mini_cluster.cc:1428] Setting key a63cea0ab19a77f68c6a2697669a215d
I20260929 16:38:21.429684 32031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a: Bootstrap starting.
Sep 29 16:38:21 dist-test-slave-3h01 krb5kdc[31951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:21.432994 32031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:21.434017 32031 log.cc:824] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:21.437067 32031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a: No bootstrap required, opened a new log
I20260929 16:38:21.441082 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:21.421010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.17.205.62:39909)
0929 16:38:21.421776 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:21.421789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:21.421835 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:38:21.423705 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:21.423728 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:21.424149 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:21.424524 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:21.425961 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:21.427330 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:21.428613 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:21.429293 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:21.429465 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:21.433488 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:21.433497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:21.433512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:21.433550 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:21.436555 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:21.437224 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:21.437232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:21.437240 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:21.437447 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:21.437797 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:21.437802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:21.437805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:21.438508 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:21.438797 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:21.439193 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 16:38:21.439687 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20260929 16:38:21.444967 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:38:21.446337 32031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } }
I20260929 16:38:21.446779 32031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:21.446866 32031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84050e0865504512ac50e725c49ec24a, State: Initialized, Role: FOLLOWER
I20260929 16:38:21.447518 32031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } }
I20260929 16:38:21.447801 32031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:21.447921 32031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:21.448086 32031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:21.450487 32031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } }
I20260929 16:38:21.450958 32031 leader_election.cc:302] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84050e0865504512ac50e725c49ec24a; no voters:
I20260929 16:38:21.451530 32031 leader_election.cc:288] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:38:21.451653 32036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:21.452301 32036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [term 1 LEADER]: Becoming Leader. State: Replica: 84050e0865504512ac50e725c49ec24a, State: Running, Role: LEADER
I20260929 16:38:21.452832 32036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } }
I20260929 16:38:21.453460 32031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:21.455669 32038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84050e0865504512ac50e725c49ec24a. Latest consensus state: current_term: 1 leader_uuid: "84050e0865504512ac50e725c49ec24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } } }
I20260929 16:38:21.456331 32038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:21.455657 32037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84050e0865504512ac50e725c49ec24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050e0865504512ac50e725c49ec24a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 39909 } } }
I20260929 16:38:21.457446 32037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:21.457896 32044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:21.462198 32044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:21.470796 32044 catalog_manager.cc:1347] Generated new cluster ID: 8f96aa5d8e4b4da28a7250454efbdcab
I20260929 16:38:21.470934 32044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:21.492113 32044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:38:21.493492 32044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:21.501861 32044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84050e0865504512ac50e725c49ec24a: Generated new TSK 0
I20260929 16:38:21.502754 32044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:38:21.558395 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 31964
I20260929 16:38:21.559617 32026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:21.656086 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 31964
2026-09-29T16:38:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1890 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:38:21 dist-test-slave-3h01 krb5kdc[32060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:38:21 dist-test-slave-3h01 krb5kdc[32060](info): set up 2 sockets
Sep 29 16:38:21 dist-test-slave-3h01 krb5kdc[32060](info): commencing operation
krb5kdc: starting...
W20260929 16:38:22.667688 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:38:22 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:38:22.890900 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:23.218806 32073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:23.219126 32073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:23.219269 32073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:23.226703 32073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:23.226835 32073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:23.226900 32073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:23.226981 32073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:23.227042 32073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:23.237218 32073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:37887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:23.238600 32073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:23.240006 32073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:23.251603 32079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:23.251806 32078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:23.251871 32081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:23.252885 32073 server_base.cc:1034] running on GCE node
I20260929 16:38:23.253674 32073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:23.255165 32073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:23.256387 32073 hybrid_clock.cc:648] HybridClock initialized: now 1790699903256331 us; error 45 us; skew 500 ppm
I20260929 16:38:23.256736 32073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:23 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699903, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:23.265712 32073 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:23.267366 32073 webserver.cc:460] Webserver started at http://127.17.205.62:44997/ using document root <none> and password file <none>
I20260929 16:38:23.268071 32073 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:23.268172 32073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:23.268431 32073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:23.271544 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "626264d0a1be65442480cd118bd17a71"
server_key_iv: "38badab909a32b25ae693823995ad6f4"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.272303 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "626264d0a1be65442480cd118bd17a71"
server_key_iv: "38badab909a32b25ae693823995ad6f4"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.277623 32073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:38:23.281224 32088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:23.282764 32073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:38:23.282956 32073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "626264d0a1be65442480cd118bd17a71"
server_key_iv: "38badab909a32b25ae693823995ad6f4"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.283147 32073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:23.301960 32073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:23.306202 32073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:23.306476 32073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:23.323869 32073 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:37887
I20260929 16:38:23.323897 32139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:37887 every 8 connection(s)
I20260929 16:38:23.325301 32073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:23.329000 32140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:23.333047 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32073
I20260929 16:38:23.333263 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:38:23.334019 18228 external_mini_cluster.cc:1428] Setting key 48484efa8b944f6e0eaae73ba1fb505b
I20260929 16:38:23.341020 32140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap starting.
I20260929 16:38:23.344790 32140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:23.345993 32140 log.cc:824] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:23.349562 32140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: No bootstrap required, opened a new log
Sep 29 16:38:23 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:23.359027 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:23.336232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.17.205.62:37887)
0929 16:38:23.337033 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:23.337045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:23.337097 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:38:23.339436 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:23.339464 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:23.339883 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:23.340240 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:23.341987 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.343430 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.345239 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.345970 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.346115 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:23.351023 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:23.351031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:23.351051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:23.351097 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:23.354675 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:23.355603 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:23.355610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:23.355617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:23.355797 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:23.356208 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:23.356212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:23.356214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:23.356821 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:23.357059 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:23.357538 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 16:38:23.357969 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260929 16:38:23.359057 32140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:23.359517 32140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:23.359613 32140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Initialized, Role: FOLLOWER
I20260929 16:38:23.360203 32140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:23.360447 32140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:23.360597 32140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:23.360790 32140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:23.363529 32140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:23.364014 32140 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef1e69903bc4e51b10e5ac2f055c9a1; no voters:
I20260929 16:38:23.364645 32140 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:38:23.364794 32145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:23.365422 32145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Running, Role: LEADER
I20260929 16:38:23.365978 32145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:23.366926 32140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:23.368805 32147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef1e69903bc4e51b10e5ac2f055c9a1. Latest consensus state: current_term: 1 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:23.369540 32147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:23.369385 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:23.369681 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:23.371258 32151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:23.376327 32151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:23.385478 32151 catalog_manager.cc:1347] Generated new cluster ID: 5aea9e5196ca41b99f505a5cff91495f
I20260929 16:38:23.385666 32151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:23.405364 32151 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:38:23.406718 32151 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:23.419924 32151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Generated new TSK 0
I20260929 16:38:23.420802 32151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:38:23.478677 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:23.810974 32166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:23.811292 32166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:23.811391 32166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:23.818733 32166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:23.818877 32166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:23.818994 32166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:23.828903 32166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:23.830193 32166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:23.831588 32166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:23.845491 32172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:23.845520 32171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:23.845566 32174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:23.846122 32166 server_base.cc:1034] running on GCE node
I20260929 16:38:23.846753 32166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:23.847784 32166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:23.849007 32166 hybrid_clock.cc:648] HybridClock initialized: now 1790699903848961 us; error 41 us; skew 500 ppm
I20260929 16:38:23.849309 32166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:23 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699903, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:23.852399 32166 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:23.853817 32166 webserver.cc:460] Webserver started at http://127.17.205.1:44173/ using document root <none> and password file <none>
I20260929 16:38:23.854530 32166 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:23.854648 32166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:23.854939 32166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:23.857918 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "2d79edb985474f7a85a6ee63d4a20f07"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "8df36b619ced95e5869116dbda0966d9"
server_key_iv: "b2dabef4b6a16de2120f02a139c3e77e"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.858683 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "2d79edb985474f7a85a6ee63d4a20f07"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "8df36b619ced95e5869116dbda0966d9"
server_key_iv: "b2dabef4b6a16de2120f02a139c3e77e"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.864259 32166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:38:23.868059 32181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:23.869719 32166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:38:23.869948 32166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2d79edb985474f7a85a6ee63d4a20f07"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "8df36b619ced95e5869116dbda0966d9"
server_key_iv: "b2dabef4b6a16de2120f02a139c3e77e"
server_key_version: "encryptionkey@0"
I20260929 16:38:23.870149 32166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:23.881497 32166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:23.885725 32166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:23.886040 32166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:23.887125 32166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:23.888551 32166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:23.888629 32166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:23.888746 32166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:23.888796 32166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:23.918674 32166 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:36603
I20260929 16:38:23.918727 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:36603 every 8 connection(s)
I20260929 16:38:23.920007 32166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:23.924046 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32166
I20260929 16:38:23.924230 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:38:23.924834 18228 external_mini_cluster.cc:1428] Setting key a7d9414bb6c7bfcfacbb3cf1f0234cf3
Sep 29 16:38:23 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699903, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:23.943131 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:23.922435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43309 (local address 127.17.205.62:37887)
0929 16:38:23.922894 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:23.922900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:23.924039 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 16:38:23.926128 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:23.926135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:23.926229 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:23.926339 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:23.928531 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.929293 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.931668 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.932136 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.932250 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:23.936742 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:23.936750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:23.936755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:23.936800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:23.939863 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:23.940671 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:23.940676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:23.940679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:23.940841 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:23.941283 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:23.941287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:23.941290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:23.941627 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:23.941764 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:23.942565 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 16:38:23.942794 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260929 16:38:23.944188 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:23.923074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:43309)
0929 16:38:23.923860 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:23.923910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:23.925834 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:23.926617 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:23.926626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:23.927417 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:23.928296 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:23.928326 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.929458 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.929464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:23.931476 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:23.931500 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:23.932396 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:23.932404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:23.932705 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:23.933496 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:23.933530 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:23.936445 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:23.940039 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:23.940047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:23.940063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:23.940543 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:23.940953 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:23.940957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:23.940959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:23.941175 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:23.941756 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:23.941763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:23.942158 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 16:38:23.942424 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 16:38:23.942887 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260929 16:38:23.946038 32294 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:23.946410 32294 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:23.947373 32294 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:38:23.949961 32105 ts_manager.cc:194] Registered new tserver with Master: 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603)
I20260929 16:38:23.951570 32105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:43309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:38:23.977484 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:24.300640 32300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:24.300930 32300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:24.301023 32300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:24.308295 32300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:24.308413 32300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:24.308524 32300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:24.318318 32300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:24.319598 32300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:24.320910 32300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:24.335374 32306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:24.335512 32308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:24.335846 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:24.336635 32300 server_base.cc:1034] running on GCE node
I20260929 16:38:24.337365 32300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:24.338474 32300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:24.339691 32300 hybrid_clock.cc:648] HybridClock initialized: now 1790699904339628 us; error 63 us; skew 500 ppm
I20260929 16:38:24.339988 32300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:24 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699904, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:24.343338 32300 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:24.344730 32300 webserver.cc:460] Webserver started at http://127.17.205.2:43351/ using document root <none> and password file <none>
I20260929 16:38:24.345468 32300 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:24.345598 32300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:24.345873 32300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:24.349203 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "6a686bd259d84bf0b893bbdbb90252c8"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "5182ec913ec40404f73cffa8f8613618"
server_key_iv: "d0ba0f9df92beea2093180ae1330616f"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.349967 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "6a686bd259d84bf0b893bbdbb90252c8"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "5182ec913ec40404f73cffa8f8613618"
server_key_iv: "d0ba0f9df92beea2093180ae1330616f"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.355316 32300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:38:24.358840 32315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.360273 32300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:38:24.360417 32300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "6a686bd259d84bf0b893bbdbb90252c8"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "5182ec913ec40404f73cffa8f8613618"
server_key_iv: "d0ba0f9df92beea2093180ae1330616f"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.360611 32300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:24.386981 32300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:24.391331 32300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:24.391587 32300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:24.392537 32300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:24.393872 32300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:24.393942 32300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.394028 32300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:24.394069 32300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.425822 32300 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:41517
I20260929 16:38:24.425910 32427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:41517 every 8 connection(s)
I20260929 16:38:24.427327 32300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:24.429319 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32300
I20260929 16:38:24.429469 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:38:24.430038 18228 external_mini_cluster.cc:1428] Setting key 7ba8c6bb14ee2e2edd16d582d24b1c32
Sep 29 16:38:24 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699904, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:24.449150 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:24.430392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:34187 (local address 127.17.205.62:37887)
0929 16:38:24.430577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:24.430582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:24.431957 (+ 1375us) server_negotiation.cc:407] Connection header received
0929 16:38:24.433959 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:24.433965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:24.434060 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:24.434160 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:24.435853 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.436602 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.438245 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.438663 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.438761 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:24.443136 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:24.443141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:24.443145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:24.443182 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:24.445843 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.446707 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.446710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.446713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.446835 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.447311 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.447314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.447316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.447597 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:24.447707 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:24.448671 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 16:38:24.448895 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:38:24.450659 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:24.430989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:34187)
0929 16:38:24.431792 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:24.431845 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:24.433664 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:24.434437 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:24.434448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:24.435001 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:24.435669 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:24.435690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.436785 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.436790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:24.438091 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:24.438113 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.438897 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.438905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:24.439258 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:24.439959 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:24.439992 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:24.442872 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:24.445988 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:24.445997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:24.446014 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:24.446599 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:24.446927 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:24.446931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:24.446935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:24.447169 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:24.447722 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:24.447731 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:24.448214 (+ 483us) client_negotiation.cc:769] Sending connection context
0929 16:38:24.448559 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 16:38:24.449163 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":349,"threads_started":1}
I20260929 16:38:24.452117 32428 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:24.452487 32428 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
I20260929 16:38:24.453382 32428 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
I20260929 16:38:24.454795 32105 ts_manager.cc:194] Registered new tserver with Master: 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
I20260929 16:38:24.455749 32105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:34187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:38:24.482023 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:24.806126 32433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:24.806418 32433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:24.806541 32433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:24.813882 32433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:24.814002 32433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:24.814145 32433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:24.823927 32433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:24.825256 32433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:24.826620 32433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:24.839522 32438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:24.839536 32439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:24.840313 32441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:24.841167 32433 server_base.cc:1034] running on GCE node
I20260929 16:38:24.841791 32433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:24.842811 32433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:24.844022 32433 hybrid_clock.cc:648] HybridClock initialized: now 1790699904843966 us; error 68 us; skew 500 ppm
I20260929 16:38:24.844339 32433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:24 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699904, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:24.847592 32433 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:24.849162 32433 webserver.cc:460] Webserver started at http://127.17.205.3:36285/ using document root <none> and password file <none>
I20260929 16:38:24.849879 32433 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:24.850003 32433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:24.850293 32433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:24.853331 32433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "e416fbebf92f4ab9960d32735c50f819"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "de97d47c1255267836f06cc3fd46e0d9"
server_key_iv: "e5d671293d89dd26f14c52c8a126fc26"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.854135 32433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "e416fbebf92f4ab9960d32735c50f819"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "de97d47c1255267836f06cc3fd46e0d9"
server_key_iv: "e5d671293d89dd26f14c52c8a126fc26"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.859618 32433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:38:24.863274 32448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.864743 32433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:38:24.864928 32433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "e416fbebf92f4ab9960d32735c50f819"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "de97d47c1255267836f06cc3fd46e0d9"
server_key_iv: "e5d671293d89dd26f14c52c8a126fc26"
server_key_version: "encryptionkey@0"
I20260929 16:38:24.865159 32433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:24.896499 32433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:24.900985 32433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:24.901258 32433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:24.902243 32433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:24.903702 32433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:24.903779 32433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.903916 32433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:24.903970 32433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:24.935525 32433 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:37983
I20260929 16:38:24.935568 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:37983 every 8 connection(s)
I20260929 16:38:24.936779 32433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:24.943769 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32433
I20260929 16:38:24.943928 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:38:24.944373 18228 external_mini_cluster.cc:1428] Setting key f4bdfe56387f0c521cda46e9d76ccaf3
Sep 29 16:38:24 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699904, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:24.954698 32294 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:24.957504 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:24.939297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:52457 (local address 127.17.205.62:37887)
0929 16:38:24.939489 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:24.939494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:24.940680 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 16:38:24.942509 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:24.942514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:24.942598 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:24.942688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:24.944342 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.945350 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.947411 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.947908 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.948003 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:24.951396 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:24.951403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:24.951408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:24.951449 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:24.954226 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.955040 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.955047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.955051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.955240 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.955738 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.955744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.955747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.956135 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:24.956321 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:24.957033 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 16:38:24.957222 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:38:24.958503 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:24.939815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:52457)
0929 16:38:24.940513 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:24.940558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:24.942216 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:24.942894 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:24.942923 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:24.943512 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:24.944159 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:24.944181 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.945526 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.945531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:24.947281 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:24.947298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.948100 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.948105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:24.948305 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:24.948918 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:24.948945 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:24.951089 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:24.954400 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:24.954409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:24.954425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:24.954905 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:24.955370 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:24.955374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:24.955378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:24.955616 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:24.956259 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:24.956267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:24.956662 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 16:38:24.956892 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:38:24.957315 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260929 16:38:24.959828 32561 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:24.960186 32561 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:24.960958 32561 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
I20260929 16:38:24.962391 32105 ts_manager.cc:194] Registered new tserver with Master: e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983)
I20260929 16:38:24.963326 32105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:52457
I20260929 16:38:24.972592 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:38:24.990047 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:24.977283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.17.205.62:37887)
0929 16:38:24.977474 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:24.977481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:24.977870 (+ 389us) server_negotiation.cc:407] Connection header received
0929 16:38:24.978052 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:24.978056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:24.978124 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:24.978205 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:24.979657 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.980384 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.981763 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:24.982181 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:24.982297 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:24.984762 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:24.984767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:24.984770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:24.984808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:24.987363 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.988108 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.988114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.988116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.988260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:24.988637 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:24.988641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:24.988643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:24.988979 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:24.989216 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:24.989531 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:38:24.989744 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:38:24.994051 32105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:38:25.000849 32105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:38:25.024474 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.012747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37983 (local address 127.0.0.1:50516)
0929 16:38:25.015052 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.015093 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:25.015410 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:25.016493 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:25.016499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:25.016529 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:25.016936 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.016951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.018927 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.018933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.021299 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.021313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.023443 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.023449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.023654 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.023712 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:38:25.023842 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:38:25.024024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"spinlock_wait_cycles":9344,"thread_start_us":2079,"threads_started":1}
I20260929 16:38:25.025360 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.012863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.17.205.3:37983)
0929 16:38:25.014568 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.014576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.015232 (+ 656us) server_negotiation.cc:407] Connection header received
0929 16:38:25.015642 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.015651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.015921 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.016106 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:25.017136 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.018724 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.021495 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.022911 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.023002 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.023253 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.024862 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.025037 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552,"thread_start_us":1407,"threads_started":1}
I20260929 16:38:25.026830 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.013357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.17.205.2:41517)
0929 16:38:25.014083 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.014095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.015226 (+ 1131us) server_negotiation.cc:407] Connection header received
0929 16:38:25.016731 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.016755 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.017021 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.017178 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:25.018053 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.020581 (+ 2528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.023011 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.025469 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.025578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.025767 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.026361 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.026536 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":387,"threads_started":1}
I20260929 16:38:25.027595 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.012792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:41517 (local address 127.0.0.1:32966)
0929 16:38:25.015058 (+ 2266us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.015093 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:25.015400 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:25.017466 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:25.017470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:25.017494 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:25.017872 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.017884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.020748 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.020753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.022820 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.022836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.025975 (+ 3139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.025982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.026162 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.026194 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:38:25.027025 (+ 831us) client_negotiation.cc:240] Negotiation successful
0929 16:38:25.027172 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2171,"spinlock_wait_cycles":6272,"thread_start_us":1806,"threads_started":1}
I20260929 16:38:25.030417 32496 tablet_service.cc:1467] Processing CreateTablet for tablet f3503e367c484bf6b8b692464936a09e (DEFAULT_TABLE table=test-table [id=e104e4f48c134ed0be1e57c221facba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:25.031985 32496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3503e367c484bf6b8b692464936a09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:25.032923 32363 tablet_service.cc:1467] Processing CreateTablet for tablet f3503e367c484bf6b8b692464936a09e (DEFAULT_TABLE table=test-table [id=e104e4f48c134ed0be1e57c221facba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:25.034374 32363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3503e367c484bf6b8b692464936a09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:25.035549 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.015572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48856)
0929 16:38:25.027870 (+ 12298us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.027888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:25.027997 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:25.029030 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:25.029034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:25.029063 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:25.029255 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.029267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.030933 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.030938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.032680 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.032704 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.034738 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.034744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.034960 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.034994 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:38:25.035081 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:38:25.035254 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12201,"thread_start_us":13928,"threads_started":1}
I20260929 16:38:25.035961 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.015960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.17.205.1:36603)
0929 16:38:25.028115 (+ 12155us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.028124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.028157 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:38:25.028266 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.028275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.028537 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.029448 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:25.029697 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.030776 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.032867 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.034270 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.034410 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.034634 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.035456 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.035626 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11958,"thread_start_us":12716,"threads_started":1}
I20260929 16:38:25.039891 32229 tablet_service.cc:1467] Processing CreateTablet for tablet f3503e367c484bf6b8b692464936a09e (DEFAULT_TABLE table=test-table [id=e104e4f48c134ed0be1e57c221facba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:25.041437 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3503e367c484bf6b8b692464936a09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:25.045882 32579 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Bootstrap starting.
I20260929 16:38:25.048978 32580 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Bootstrap starting.
I20260929 16:38:25.049873 32579 tablet_bootstrap.cc:654] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:25.051189 32579 log.cc:824] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:25.052856 32580 tablet_bootstrap.cc:654] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:25.054065 32580 log.cc:824] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:25.054735 32579 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: No bootstrap required, opened a new log
I20260929 16:38:25.055073 32579 ts_tablet_manager.cc:1397] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 16:38:25.056048 32583 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Bootstrap starting.
I20260929 16:38:25.057120 32580 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: No bootstrap required, opened a new log
I20260929 16:38:25.057386 32580 ts_tablet_manager.cc:1397] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 16:38:25.058668 32583 tablet_bootstrap.cc:654] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:25.059649 32583 log.cc:824] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:25.062253 32583 tablet_bootstrap.cc:492] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: No bootstrap required, opened a new log
I20260929 16:38:25.062587 32583 ts_tablet_manager.cc:1397] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:38:25.065340 32579 raft_consensus.cc:348] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.065786 32579 raft_consensus.cc:374] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:25.065888 32579 raft_consensus.cc:729] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a686bd259d84bf0b893bbdbb90252c8, State: Initialized, Role: FOLLOWER
I20260929 16:38:25.066512 32579 consensus_queue.cc:260] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.067672 32428 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:25.067636 32580 raft_consensus.cc:348] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.068037 32580 raft_consensus.cc:374] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:25.068127 32580 raft_consensus.cc:729] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e416fbebf92f4ab9960d32735c50f819, State: Initialized, Role: FOLLOWER
I20260929 16:38:25.068414 32579 ts_tablet_manager.cc:1428] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:38:25.068847 32580 consensus_queue.cc:260] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.069922 32561 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:25.070961 32583 raft_consensus.cc:348] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.071492 32583 raft_consensus.cc:374] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:25.071594 32583 raft_consensus.cc:729] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d79edb985474f7a85a6ee63d4a20f07, State: Initialized, Role: FOLLOWER
I20260929 16:38:25.072090 32580 ts_tablet_manager.cc:1428] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 16:38:25.072171 32583 consensus_queue.cc:260] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.073941 32583 ts_tablet_manager.cc:1428] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
W20260929 16:38:25.172262 32295 tablet.cc:2369] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:38:25.179777 32429 tablet.cc:2369] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:38:25.188270 32562 tablet.cc:2369] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:25.317962 32587 raft_consensus.cc:482] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:25.318272 32587 raft_consensus.cc:504] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.320066 32587 leader_election.cc:288] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983), 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517)
I20260929 16:38:25.327458 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.320543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:57265 (local address 127.17.205.2:41517)
0929 16:38:25.320771 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.320778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.321525 (+ 747us) server_negotiation.cc:407] Connection header received
0929 16:38:25.321669 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.321673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.321732 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.321810 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:25.322465 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.323167 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.325766 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.326548 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.326690 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.326765 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.327036 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.327163 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:38:25.327565 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.320529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:41517 (local address 127.17.205.1:57265)
0929 16:38:25.321426 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.321440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:25.321551 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:25.321840 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:25.321844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:25.321944 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:25.322299 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.322307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.323332 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.323336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.325566 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.325582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.326746 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.326752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.326892 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.326916 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:38:25.326996 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:38:25.327128 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":22,"spinlock_wait_cycles":24960,"thread_start_us":423,"threads_started":1}
I20260929 16:38:25.328110 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.320508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:51455 (local address 127.17.205.3:37983)
0929 16:38:25.320771 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.320779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.321433 (+ 654us) server_negotiation.cc:407] Connection header received
0929 16:38:25.321651 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.321657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.321722 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.321793 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:25.322439 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.323148 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.325732 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.326956 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.327118 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.327186 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.327737 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.327871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 16:38:25.328656 32383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3503e367c484bf6b8b692464936a09e" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a686bd259d84bf0b893bbdbb90252c8" is_pre_election: true
I20260929 16:38:25.328842 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.320490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37983 (local address 127.17.205.1:51455)
0929 16:38:25.321321 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.321336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:25.321482 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:25.321831 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:25.321835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:25.321858 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:25.322297 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.322306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.323272 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.323276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.325574 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:25.325582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.327362 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.327368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:25.327564 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.327597 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:38:25.328336 (+ 739us) client_negotiation.cc:240] Negotiation successful
0929 16:38:25.328482 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":33,"spinlock_wait_cycles":30592,"thread_start_us":553,"threads_started":1}
I20260929 16:38:25.329192 32383 raft_consensus.cc:2454] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d79edb985474f7a85a6ee63d4a20f07 in term 0.
I20260929 16:38:25.329773 32516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3503e367c484bf6b8b692464936a09e" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416fbebf92f4ab9960d32735c50f819" is_pre_election: true
I20260929 16:38:25.329900 32184 leader_election.cc:302] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d79edb985474f7a85a6ee63d4a20f07, 6a686bd259d84bf0b893bbdbb90252c8; no voters:
I20260929 16:38:25.330305 32587 raft_consensus.cc:2781] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:38:25.330283 32516 raft_consensus.cc:2454] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d79edb985474f7a85a6ee63d4a20f07 in term 0.
I20260929 16:38:25.330440 32587 raft_consensus.cc:482] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:38:25.330538 32587 raft_consensus.cc:3037] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:25.332825 32587 raft_consensus.cc:504] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.333441 32587 leader_election.cc:288] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 election: Requested vote from peers e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983), 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517)
I20260929 16:38:25.333875 32516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3503e367c484bf6b8b692464936a09e" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416fbebf92f4ab9960d32735c50f819"
I20260929 16:38:25.333966 32383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3503e367c484bf6b8b692464936a09e" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a686bd259d84bf0b893bbdbb90252c8"
I20260929 16:38:25.334081 32516 raft_consensus.cc:3037] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:25.334160 32383 raft_consensus.cc:3037] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:25.337174 32383 raft_consensus.cc:2454] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d79edb985474f7a85a6ee63d4a20f07 in term 1.
I20260929 16:38:25.337174 32516 raft_consensus.cc:2454] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d79edb985474f7a85a6ee63d4a20f07 in term 1.
I20260929 16:38:25.337555 32182 leader_election.cc:302] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d79edb985474f7a85a6ee63d4a20f07, e416fbebf92f4ab9960d32735c50f819; no voters:
I20260929 16:38:25.337858 32587 raft_consensus.cc:2781] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:25.338296 32587 raft_consensus.cc:686] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 LEADER]: Becoming Leader. State: Replica: 2d79edb985474f7a85a6ee63d4a20f07, State: Running, Role: LEADER
I20260929 16:38:25.338832 32587 consensus_queue.cc:237] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:25.343281 32104 catalog_manager.cc:5495] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "2d79edb985474f7a85a6ee63d4a20f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } health_report { overall_health: HEALTHY } } }
I20260929 16:38:25.389060 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.383871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.17.205.1:36603)
0929 16:38:25.384076 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.384081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.384100 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:38:25.384304 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.384308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.384404 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.384529 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:38:25.384920 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.385619 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.386981 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.387417 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.387528 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.388030 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:38:25.388518 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:38:25.388674 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.388725 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.388805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:38:25.396845 32383 raft_consensus.cc:1261] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Refusing update from remote peer 2d79edb985474f7a85a6ee63d4a20f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:25.396849 32516 raft_consensus.cc:1261] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Refusing update from remote peer 2d79edb985474f7a85a6ee63d4a20f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:25.397745 32590 consensus_queue.cc:1035] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:25.398056 32587 consensus_queue.cc:1035] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:25.408975 32595 mvcc.cc:205] Tried to move back new op lower bound from 7334706812496166912 to 7334706812271857664. Current Snapshot: MvccSnapshot[applied={T|T < 7334706812496166912}]
I20260929 16:38:25.423805 32104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:25.424121 32104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:25.428260 32104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Sending DeleteTablet for 3 replicas of tablet f3503e367c484bf6b8b692464936a09e
I20260929 16:38:25.429474 32496 tablet_service.cc:1514] Processing DeleteTablet for tablet f3503e367c484bf6b8b692464936a09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:25 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260929 16:38:25.429741 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32166
I20260929 16:38:25.429854 32363 tablet_service.cc:1514] Processing DeleteTablet for tablet f3503e367c484bf6b8b692464936a09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:25 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260929 16:38:25.429841 32229 tablet_service.cc:1514] Processing DeleteTablet for tablet f3503e367c484bf6b8b692464936a09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:25 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:48856
I20260929 16:38:25.442391 32603 tablet_replica.cc:330] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: stopping tablet replica
I20260929 16:38:25.443830 32603 raft_consensus.cc:2229] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:25.446611 32603 raft_consensus.cc:2258] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:25.447701 32604 tablet_replica.cc:330] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: stopping tablet replica
I20260929 16:38:25.448172 32604 raft_consensus.cc:2229] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:25.449229 32604 raft_consensus.cc:2258] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:25.450241 32605 tablet_replica.cc:330] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: stopping tablet replica
I20260929 16:38:25.450649 32605 raft_consensus.cc:2229] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:38:25.451362 32605 raft_consensus.cc:2258] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:25.451942 32603 ts_tablet_manager.cc:1905] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:25.453691 32605 ts_tablet_manager.cc:1905] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:25.455276 32604 ts_tablet_manager.cc:1905] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:38:25 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:38:25.462808 32605 ts_tablet_manager.cc:1918] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:25.465912 32604 ts_tablet_manager.cc:1918] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:25.466269 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.430876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.17.205.1:36603)
0929 16:38:25.431067 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.431074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.443033 (+ 11959us) server_negotiation.cc:407] Connection header received
0929 16:38:25.444137 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.444143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.444252 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.444328 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:25.447056 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.448237 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.451766 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.452507 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.452606 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.458102 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:25.458116 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:25.458132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:25.458186 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:25.462356 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.463370 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.463380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.463384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.463557 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.464090 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.464095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.464098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.464587 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:25.464980 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.465610 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.465866 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:38:25.467181 32605 log.cc:1196] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/f3503e367c484bf6b8b692464936a09e
I20260929 16:38:25.467717 32289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:25.467969 32604 log.cc:1196] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/f3503e367c484bf6b8b692464936a09e
I20260929 16:38:25.468565 32604 ts_tablet_manager.cc:1939] T f3503e367c484bf6b8b692464936a09e P 6a686bd259d84bf0b893bbdbb90252c8: Deleting consensus metadata
I20260929 16:38:25.468699 32603 ts_tablet_manager.cc:1918] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:25.468946 32603 log.cc:1196] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/f3503e367c484bf6b8b692464936a09e
I20260929 16:38:25.469488 32603 ts_tablet_manager.cc:1939] T f3503e367c484bf6b8b692464936a09e P e416fbebf92f4ab9960d32735c50f819: Deleting consensus metadata
I20260929 16:38:25.470539 32091 catalog_manager.cc:4841] TS 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517): tablet f3503e367c484bf6b8b692464936a09e (table test-table [id=e104e4f48c134ed0be1e57c221facba7]) successfully deleted
I20260929 16:38:25.471449 32089 catalog_manager.cc:4841] TS e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983): tablet f3503e367c484bf6b8b692464936a09e (table test-table [id=e104e4f48c134ed0be1e57c221facba7]) successfully deleted
I20260929 16:38:25.566398 32605 ts_tablet_manager.cc:1939] T f3503e367c484bf6b8b692464936a09e P 2d79edb985474f7a85a6ee63d4a20f07: Deleting consensus metadata
I20260929 16:38:25.567076 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32166
I20260929 16:38:25.582368 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32300
W20260929 16:38:25.582361 32091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:36603 (error 108)
I20260929 16:38:25.584371 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.583577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48866)
0929 16:38:25.584020 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.584160 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":186,"threads_started":1}
W20260929 16:38:25.584620 32091 catalog_manager.cc:4584] TS 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3503e367c484bf6b8b692464936a09e: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Sep 29 16:38:25 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:38:25.597143 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.583097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.17.205.2:41517)
0929 16:38:25.583556 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.583562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.583581 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:38:25.583650 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.583655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.583743 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.583858 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:25.585341 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.586075 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.587505 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.587973 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.588089 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.592128 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:25.592144 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:25.592160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:25.592195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:25.594621 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.595290 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.595298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.595300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.595441 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.595735 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.595737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.595739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.596124 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:25.596345 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.596665 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.596834 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260929 16:38:25.597720 32423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:25.638262 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.637801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48868)
0929 16:38:25.638011 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.638103 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:38:25.639442 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.639026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48872)
0929 16:38:25.639178 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.639302 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:38:25.688514 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32300
I20260929 16:38:25.700094 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.699549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48878)
0929 16:38:25.699779 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.699896 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:38:25.701136 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.700741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48894)
0929 16:38:25.700869 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.700958 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 16:38:25.702629 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32433
Sep 29 16:38:25 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:38:25.716898 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.703403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.17.205.3:37983)
0929 16:38:25.703577 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:25.703581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:25.703599 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:38:25.703691 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:25.703695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:25.703764 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:25.703843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:25.705099 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.705789 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.707074 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:25.707526 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:25.707622 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:25.711884 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:25.711901 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:25.711930 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:25.711966 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:25.714416 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.715033 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.715040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.715042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.715167 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:25.715598 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:25.715601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:25.715603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:25.715966 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:25.716187 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:25.716418 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:38:25.716589 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:38:25.717545 32556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:25.781601 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.781110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48898)
0929 16:38:25.781353 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.781454 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:38:25.782634 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:25.782206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48914)
0929 16:38:25.782394 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:25.782476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:38:25.804641 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32433
I20260929 16:38:25.818043 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32073
I20260929 16:38:25.819123 32135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:25.929673 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32073
I20260929 16:38:25.947081 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--webserver_interface=127.17.205.62
--webserver_port=44997
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:26.264120 32613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:26.264446 32613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:26.264583 32613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:26.271806 32613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:26.271929 32613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:26.272006 32613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:26.272078 32613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:26.272136 32613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:26.281987 32613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:37887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:26.283387 32613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:26.284683 32613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:26.305465 32622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:26.305534 32619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:26.305487 32620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:26.306244 32613 server_base.cc:1034] running on GCE node
I20260929 16:38:26.307093 32613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:26.308703 32613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:26.309926 32613 hybrid_clock.cc:648] HybridClock initialized: now 1790699906309877 us; error 39 us; skew 500 ppm
I20260929 16:38:26.310325 32613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:26 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699906, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:26.313730 32613 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:26.315500 32613 webserver.cc:460] Webserver started at http://127.17.205.62:44997/ using document root <none> and password file <none>
I20260929 16:38:26.316260 32613 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:26.316391 32613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:26.322413 32613 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 16:38:26.325562 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:26.327013 32613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:38:26.327169 32613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "626264d0a1be65442480cd118bd17a71"
server_key_iv: "38badab909a32b25ae693823995ad6f4"
server_key_version: "encryptionkey@0"
I20260929 16:38:26.328258 32613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:26.344471 32613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:26.348829 32613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:26.349114 32613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:26.366901 32613 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:37887
I20260929 16:38:26.366953 32680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:37887 every 8 connection(s)
I20260929 16:38:26.368583 32613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:26.375886 32681 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:38:26.378001 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32613
I20260929 16:38:26.378754 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:36603
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=44173
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:26.379380 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap starting.
I20260929 16:38:26.398847 32681 log.cc:824] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:26.411137 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:38:26.411697 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap complete.
I20260929 16:38:26.420665 32681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:26.421541 32681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Initialized, Role: FOLLOWER
I20260929 16:38:26.422137 32681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:26.422375 32681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:26.422499 32681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:26.422694 32681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:38:26.426504 32681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:26.427021 32681 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef1e69903bc4e51b10e5ac2f055c9a1; no voters:
I20260929 16:38:26.427668 32681 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:38:26.427799 32685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:38:26.428824 32685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 LEADER]: Becoming Leader. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Running, Role: LEADER
I20260929 16:38:26.429462 32685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:26.429945 32681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:26.431926 32687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef1e69903bc4e51b10e5ac2f055c9a1. Latest consensus state: current_term: 2 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:26.432303 32687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:26.432950 32686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:26.433231 32686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:26.440382 32690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:26.449404 32690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:26.451020 32690 catalog_manager.cc:1259] Loaded cluster ID: 5aea9e5196ca41b99f505a5cff91495f
I20260929 16:38:26.451225 32690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:26.454720 32690 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:38:26.474025 32690 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:38:26.475642 32702 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:38:26.481688 32690 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:26.483108 32690 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Loaded TSK: 0
I20260929 16:38:26.484298 32690 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:38:26.737481 32683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:26.737747 32683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:26.737861 32683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:26.745214 32683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:26.745319 32683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:26.745464 32683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:26.755031 32683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:36603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=44173
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:26.756276 32683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:26.757575 32683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:26.771344 32709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:26.771306 32708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:26.771950 32711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:26.772172 32683 server_base.cc:1034] running on GCE node
I20260929 16:38:26.772769 32683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:26.773698 32683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:26.774900 32683 hybrid_clock.cc:648] HybridClock initialized: now 1790699906774846 us; error 46 us; skew 500 ppm
I20260929 16:38:26.775189 32683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:26 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699906, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:26.778302 32683 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:26.779894 32683 webserver.cc:460] Webserver started at http://127.17.205.1:44173/ using document root <none> and password file <none>
I20260929 16:38:26.780558 32683 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:26.780674 32683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:26.785953 32683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:38:26.788637 32718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:26.789999 32683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:38:26.790148 32683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2d79edb985474f7a85a6ee63d4a20f07"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "8df36b619ced95e5869116dbda0966d9"
server_key_iv: "b2dabef4b6a16de2120f02a139c3e77e"
server_key_version: "encryptionkey@0"
I20260929 16:38:26.791061 32683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:26.806171 32683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:26.810106 32683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:26.810345 32683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:26.811456 32683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:26.812759 32683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:26.812850 32683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:26.812949 32683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:26.813024 32683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:26.842860 32683 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:36603
I20260929 16:38:26.842931 362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:36603 every 8 connection(s)
I20260929 16:38:26.844390 32683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:26.850191 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 32683
I20260929 16:38:26.850762 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:41517
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=43351
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:26 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699906, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:26.870280 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:26.847904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:47263 (local address 127.17.205.62:37887)
0929 16:38:26.848456 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:26.848468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:26.849164 (+ 696us) server_negotiation.cc:407] Connection header received
0929 16:38:26.851524 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:26.851545 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:26.852044 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:26.852458 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:26.854002 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:26.855834 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:26.857536 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:26.858256 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:26.858451 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:26.862285 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:26.862294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:26.862314 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:26.862358 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:26.865399 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:26.866212 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:26.866220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:26.866226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:26.866416 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:26.866902 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:26.866907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:26.866909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:26.867492 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:26.867679 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:26.868525 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 16:38:26.868953 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260929 16:38:26.870273 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:26.848157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:47263)
0929 16:38:26.848986 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:26.849041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:26.850883 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:26.852646 (+ 1763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:26.852657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:26.853195 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:26.853761 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:26.853779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:26.856029 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:26.856034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:26.857360 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:26.857374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:26.858508 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:26.858517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:26.858794 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:26.859558 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:26.859584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:26.861969 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:26.865580 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:26.865591 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:26.865606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:26.866062 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:26.866551 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:26.866556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:26.866560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:26.866782 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:26.867645 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:26.867653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:26.868019 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 16:38:26.868285 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 16:38:26.868900 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260929 16:38:26.873625 363 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:26.873947 363 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:26.874804 363 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
I20260929 16:38:26.877122 32646 ts_manager.cc:194] Registered new tserver with Master: 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603)
I20260929 16:38:26.878764 32646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:47263
W20260929 16:38:27.182876 367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:27.183167 367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:27.183329 367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:27.190559 367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:27.190683 367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:27.190850 367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:27.200774 367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=43351
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:27.202103 367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:27.203518 367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:27.216138 372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:27.216085 373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:27.216312 375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:27.217132 367 server_base.cc:1034] running on GCE node
I20260929 16:38:27.217705 367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:27.218638 367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:27.219841 367 hybrid_clock.cc:648] HybridClock initialized: now 1790699907219781 us; error 50 us; skew 500 ppm
I20260929 16:38:27.220141 367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:27 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699907, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:27.223084 367 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:27.224491 367 webserver.cc:460] Webserver started at http://127.17.205.2:43351/ using document root <none> and password file <none>
I20260929 16:38:27.225157 367 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:27.225286 367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:27.230581 367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:38:27.233311 382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.234579 367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:38:27.234726 367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "6a686bd259d84bf0b893bbdbb90252c8"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "5182ec913ec40404f73cffa8f8613618"
server_key_iv: "d0ba0f9df92beea2093180ae1330616f"
server_key_version: "encryptionkey@0"
I20260929 16:38:27.235716 367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:27.258059 367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:27.262187 367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:27.262460 367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:27.263540 367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:27.264931 367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:27.265026 367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.265136 367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:27.265202 367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.295325 367 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:41517
I20260929 16:38:27.295424 494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:41517 every 8 connection(s)
I20260929 16:38:27.296758 367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:27.304330 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 367
I20260929 16:38:27.304905 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:37983
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=36285
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:38:27 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699907, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:27.316520 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.299180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:46461 (local address 127.17.205.62:37887)
0929 16:38:27.299374 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.299379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.300572 (+ 1193us) server_negotiation.cc:407] Connection header received
0929 16:38:27.302407 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.302413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.302494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.302594 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:27.304156 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.304870 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.306426 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.306847 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.306958 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.310426 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:27.310431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:27.310434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:27.310468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:27.313368 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.314084 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.314089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.314093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.314240 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.314623 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.314628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.314631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.314971 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:27.315112 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.315914 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.316157 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:38:27.317378 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.299712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:46461)
0929 16:38:27.300419 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.300457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.302122 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.302867 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.302876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.303422 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:27.303992 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.304009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.305020 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.305025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.306275 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.306287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.307069 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.307077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.307334 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.307977 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:27.308004 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:27.310214 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:27.313510 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:27.313517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:27.313530 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:27.313961 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:27.314328 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:27.314331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:27.314333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:27.314523 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:27.315121 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:27.315128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:27.315537 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.315782 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.316224 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":267,"threads_started":1}
I20260929 16:38:27.318970 495 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:27.319303 495 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:27.320060 495 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
I20260929 16:38:27.321607 32646 ts_manager.cc:194] Registered new tserver with Master: 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517)
I20260929 16:38:27.322428 32646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:46461
W20260929 16:38:27.625384 498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:27.625654 498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:27.625785 498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:27.633108 498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:27.633220 498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:27.633369 498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:27.643143 498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:37983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=36285
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:27.644467 498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:27.645754 498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:27.658464 503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:27.658528 504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:27.658495 506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:27.659178 498 server_base.cc:1034] running on GCE node
I20260929 16:38:27.659813 498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:27.660820 498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:27.662050 498 hybrid_clock.cc:648] HybridClock initialized: now 1790699907662002 us; error 41 us; skew 500 ppm
I20260929 16:38:27.662379 498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:27 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699907, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:27.665486 498 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:27.667479 498 webserver.cc:460] Webserver started at http://127.17.205.3:36285/ using document root <none> and password file <none>
I20260929 16:38:27.668402 498 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:27.668593 498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:27.675546 498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 16:38:27.678293 513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.679704 498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:38:27.679873 498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "e416fbebf92f4ab9960d32735c50f819"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "de97d47c1255267836f06cc3fd46e0d9"
server_key_iv: "e5d671293d89dd26f14c52c8a126fc26"
server_key_version: "encryptionkey@0"
I20260929 16:38:27.680804 498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:27.701540 498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:27.705610 498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:27.705869 498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:27.706874 498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:27.708276 498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:27.708361 498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.708463 498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:27.708530 498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:27.738318 498 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:37983
I20260929 16:38:27.738374 625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:37983 every 8 connection(s)
I20260929 16:38:27.739899 498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:27.747601 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 498
Sep 29 16:38:27 dist-test-slave-3h01 krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699907, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:27.760380 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.742417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:50729 (local address 127.17.205.62:37887)
0929 16:38:27.742604 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.742609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.743873 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 16:38:27.745688 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.745694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.745779 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.745865 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:27.747413 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.748191 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.750186 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.750824 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.750965 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.754518 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:27.754522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:27.754526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:27.754552 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:27.757027 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.757955 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.757959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.757962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.758094 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.758559 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.758563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.758565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.758833 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:27.758944 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.759835 (+ 891us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.760045 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:38:27.761410 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.743000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:50729)
0929 16:38:27.743715 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.743759 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.745425 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.746119 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.746128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.746631 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:27.747242 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.747266 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.748424 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.748428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.750044 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.750056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.751028 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.751033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.751286 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.752019 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:27.752055 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:27.754311 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:27.757160 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:27.757171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:27.757194 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:27.757815 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:27.758202 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:27.758208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:27.758211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:27.758440 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:27.758955 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:27.758963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:27.759419 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.759727 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.760136 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260929 16:38:27.762526 629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.748481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.17.205.62:37887)
0929 16:38:27.748982 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.748988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.749265 (+ 277us) server_negotiation.cc:407] Connection header received
0929 16:38:27.749428 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.749433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.749510 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.749605 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:27.751266 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.752251 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.753635 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.754059 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.754154 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.756704 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:27.756710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:27.756713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:27.756752 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:27.759621 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.760412 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.760417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.760420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.760547 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.760982 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.760986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.760989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.761306 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:27.761555 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.762051 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.762246 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260929 16:38:27.762931 626 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37887
I20260929 16:38:27.763265 626 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:27.764113 626 heartbeater.cc:511] Master 127.17.205.62:37887 requested a full tablet report, sending...
I20260929 16:38:27.765537 32646 ts_manager.cc:194] Registered new tserver with Master: e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983)
I20260929 16:38:27.766271 32646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:50729
I20260929 16:38:27.777195 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:38:27.783404 629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.778027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.17.205.62:37887)
0929 16:38:27.778191 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.778196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.778605 (+ 409us) server_negotiation.cc:407] Connection header received
0929 16:38:27.778793 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.778797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.778953 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.779079 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:38:27.779473 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.780132 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.781550 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.781959 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.782043 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.782458 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:38:27.782947 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:38:27.783051 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.783090 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.783150 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:38:27.784260 32646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:38:27.791304 32646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:38:27.813190 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.802201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:41517 (local address 127.0.0.1:32990)
0929 16:38:27.803666 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.803710 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.803931 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.804687 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.804696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.804728 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.805182 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.805195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.807511 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.807516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.810068 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.810083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.812160 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.812166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.812382 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.812442 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.812561 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.812778 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":1551,"threads_started":1}
I20260929 16:38:27.813848 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.802895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.17.205.3:37983)
0929 16:38:27.804527 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.804534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.804565 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:38:27.804654 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.804661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.804899 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.805092 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.805986 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.807729 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.810495 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.811961 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.812092 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.812340 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.813293 (+ 953us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.813488 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1298,"threads_started":1}
I20260929 16:38:27.814709 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.802475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.17.205.2:41517)
0929 16:38:27.803192 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.803201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.803815 (+ 614us) server_negotiation.cc:407] Connection header received
0929 16:38:27.804154 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.804162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.804440 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.805414 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.805700 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.807330 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.810293 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.811917 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.813839 (+ 1922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.814076 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.814146 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.814321 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":383,"threads_started":1}
I20260929 16:38:27.815093 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.802557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37983 (local address 127.0.0.1:50524)
0929 16:38:27.803665 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.803707 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.803858 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.805380 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.805386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.805412 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.805785 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.805796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.807949 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.807954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.810286 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.810303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.812520 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.812528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.812737 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.812773 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.812862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.813008 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":661,"threads_started":1}
I20260929 16:38:27.818893 430 tablet_service.cc:1467] Processing CreateTablet for tablet 7146d65421c348cd9430806668dad8a1 (DEFAULT_TABLE table=test-table [id=3323d785a3d8467f938c4c1123451cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:27.818965 561 tablet_service.cc:1467] Processing CreateTablet for tablet 7146d65421c348cd9430806668dad8a1 (DEFAULT_TABLE table=test-table [id=3323d785a3d8467f938c4c1123451cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:27.820336 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146d65421c348cd9430806668dad8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:27.820343 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146d65421c348cd9430806668dad8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:27.822281 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.804153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48920)
0929 16:38:27.813556 (+ 9403us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.813576 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.813682 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.815926 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.815931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.815954 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.816086 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.816093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.818027 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.818032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.819796 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.819810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.821560 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.821566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.821745 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.821777 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.821879 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.822028 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9301,"thread_start_us":11156,"threads_started":1}
I20260929 16:38:27.822378 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.804605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.17.205.1:36603)
0929 16:38:27.815028 (+ 10423us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.815037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.815067 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:38:27.815164 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.815173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.815714 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.816217 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.816440 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.817874 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.820010 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.821373 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.821510 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.821707 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.821941 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.822104 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10249,"thread_start_us":10138,"threads_started":1}
I20260929 16:38:27.825897 32766 tablet_service.cc:1467] Processing CreateTablet for tablet 7146d65421c348cd9430806668dad8a1 (DEFAULT_TABLE table=test-table [id=3323d785a3d8467f938c4c1123451cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:27.827350 32766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146d65421c348cd9430806668dad8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:27.832937 642 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Bootstrap starting.
I20260929 16:38:27.835569 642 tablet_bootstrap.cc:654] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:27.836438 643 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Bootstrap starting.
I20260929 16:38:27.836830 642 log.cc:824] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:27.840073 643 tablet_bootstrap.cc:654] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:27.840103 642 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: No bootstrap required, opened a new log
I20260929 16:38:27.840364 642 ts_tablet_manager.cc:1397] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 16:38:27.841287 643 log.cc:824] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:27.849447 646 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Bootstrap starting.
I20260929 16:38:27.851044 642 raft_consensus.cc:348] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.851524 642 raft_consensus.cc:374] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:27.851619 642 raft_consensus.cc:729] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e416fbebf92f4ab9960d32735c50f819, State: Initialized, Role: FOLLOWER
I20260929 16:38:27.852326 642 consensus_queue.cc:260] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.856069 626 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:27.856315 646 tablet_bootstrap.cc:654] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:27.856887 642 ts_tablet_manager.cc:1428] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260929 16:38:27.857791 646 log.cc:824] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:27.859694 643 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: No bootstrap required, opened a new log
I20260929 16:38:27.860018 643 ts_tablet_manager.cc:1397] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20260929 16:38:27.861045 646 tablet_bootstrap.cc:492] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: No bootstrap required, opened a new log
I20260929 16:38:27.861285 646 ts_tablet_manager.cc:1397] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 16:38:27.867777 643 raft_consensus.cc:348] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.868156 643 raft_consensus.cc:374] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:27.868252 643 raft_consensus.cc:729] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a686bd259d84bf0b893bbdbb90252c8, State: Initialized, Role: FOLLOWER
I20260929 16:38:27.868268 646 raft_consensus.cc:348] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.868624 646 raft_consensus.cc:374] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:27.868716 646 raft_consensus.cc:729] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d79edb985474f7a85a6ee63d4a20f07, State: Initialized, Role: FOLLOWER
I20260929 16:38:27.868865 643 consensus_queue.cc:260] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.869264 646 consensus_queue.cc:260] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.870002 495 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:27.870457 643 ts_tablet_manager.cc:1428] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260929 16:38:27.871644 363 heartbeater.cc:503] Master 127.17.205.62:37887 was elected leader, sending a full tablet report...
I20260929 16:38:27.872303 646 ts_tablet_manager.cc:1428] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:38:27.875082 649 raft_consensus.cc:482] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:27.875348 649 raft_consensus.cc:504] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.876863 649 leader_election.cc:288] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983), 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603)
I20260929 16:38:27.878213 650 raft_consensus.cc:482] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:27.878482 650 raft_consensus.cc:504] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.885149 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.877624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:35683 (local address 127.17.205.1:36603)
0929 16:38:27.877944 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.877949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.877965 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:38:27.878025 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.878028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.878085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.878172 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.878792 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.879718 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.882282 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.883783 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.883919 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.884002 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.884661 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.884824 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260929 16:38:27.886049 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.877157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.17.205.2:35683)
0929 16:38:27.877735 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.877759 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.877895 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.878209 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.878214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.878242 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.878647 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.878657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.879853 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.879856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.882074 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.882091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.884256 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.884262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.884461 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.884497 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.885414 (+ 917us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.885593 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":31,"spinlock_wait_cycles":35584,"thread_start_us":460,"threads_started":1}
I20260929 16:38:27.887225 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.877541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33609 (local address 127.17.205.3:37983)
0929 16:38:27.877737 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.877744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.878361 (+ 617us) server_negotiation.cc:407] Connection header received
0929 16:38:27.878552 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.878557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.878630 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.878718 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.879303 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.880302 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.882768 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.884079 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.884230 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.884301 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.886770 (+ 2469us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.886937 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:38:27.888224 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.877161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37983 (local address 127.17.205.2:33609)
0929 16:38:27.878265 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.878282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.878424 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.878744 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.878748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.878772 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.879130 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.879141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.880463 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.880468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.882592 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.882608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.886384 (+ 3776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.886391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.886584 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.886618 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.887692 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.887842 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":31,"thread_start_us":626,"threads_started":1}
I20260929 16:38:27.889554 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "6a686bd259d84bf0b893bbdbb90252c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416fbebf92f4ab9960d32735c50f819" is_pre_election: true
I20260929 16:38:27.890188 581 raft_consensus.cc:2454] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a686bd259d84bf0b893bbdbb90252c8 in term 0.
I20260929 16:38:27.890311 650 leader_election.cc:288] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517), e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983)
I20260929 16:38:27.890985 383 leader_election.cc:302] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a686bd259d84bf0b893bbdbb90252c8, e416fbebf92f4ab9960d32735c50f819; no voters:
I20260929 16:38:27.891479 649 raft_consensus.cc:2781] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:38:27.891701 649 raft_consensus.cc:482] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:38:27.891839 649 raft_consensus.cc:3037] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:27.895098 649 raft_consensus.cc:504] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.896214 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "6a686bd259d84bf0b893bbdbb90252c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416fbebf92f4ab9960d32735c50f819"
I20260929 16:38:27.896353 649 leader_election.cc:288] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [CANDIDATE]: Term 1 election: Requested vote from peers e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983), 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603)
I20260929 16:38:27.896445 581 raft_consensus.cc:3037] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:27.896916 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "6a686bd259d84bf0b893bbdbb90252c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d79edb985474f7a85a6ee63d4a20f07"
I20260929 16:38:27.897305 318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "6a686bd259d84bf0b893bbdbb90252c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d79edb985474f7a85a6ee63d4a20f07" is_pre_election: true
I20260929 16:38:27.897307 317 raft_consensus.cc:3037] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:27.901496 317 raft_consensus.cc:2454] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a686bd259d84bf0b893bbdbb90252c8 in term 1.
I20260929 16:38:27.901892 385 leader_election.cc:302] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d79edb985474f7a85a6ee63d4a20f07, 6a686bd259d84bf0b893bbdbb90252c8; no voters:
I20260929 16:38:27.902361 649 raft_consensus.cc:2781] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:27.903169 581 raft_consensus.cc:2454] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a686bd259d84bf0b893bbdbb90252c8 in term 1.
I20260929 16:38:27.903573 649 raft_consensus.cc:686] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6a686bd259d84bf0b893bbdbb90252c8, State: Running, Role: LEADER
I20260929 16:38:27.904165 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.881271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:53297 (local address 127.17.205.3:37983)
0929 16:38:27.887452 (+ 6181us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.887458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.896116 (+ 8658us) server_negotiation.cc:407] Connection header received
0929 16:38:27.896953 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.896959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.897033 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.897092 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.897837 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.898891 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.901314 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.902946 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.903094 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.903170 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.903742 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.903878 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6076,"thread_start_us":13086,"threads_started":1}
I20260929 16:38:27.904131 649 consensus_queue.cc:237] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } }
I20260929 16:38:27.904845 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.880643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:37983 (local address 127.17.205.1:53297)
0929 16:38:27.896015 (+ 15372us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.896041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.896803 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.897263 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.897268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.897297 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.897671 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.897681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.899025 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.899030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.901133 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.901147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.903404 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.903411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.903573 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.903608 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.904327 (+ 719us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.904468 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15251,"mutex_wait_us":3,"thread_start_us":14038,"threads_started":1}
I20260929 16:38:27.904914 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.881035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:46025 (local address 127.17.205.2:41517)
0929 16:38:27.881655 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.881661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.895249 (+ 13588us) server_negotiation.cc:407] Connection header received
0929 16:38:27.895617 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.895622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.895699 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.895783 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:27.896773 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.897747 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.900359 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.901554 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.901701 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.901774 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.904533 (+ 2759us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.904669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480}
I20260929 16:38:27.905385 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.880497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:41517 (local address 127.17.205.1:46025)
0929 16:38:27.895089 (+ 14592us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:27.895113 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:27.895458 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:27.896184 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:27.896188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:27.896215 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:27.896632 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.896646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.897850 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.897855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.900203 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:27.900217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.904110 (+ 3893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.904116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:27.904382 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.904412 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:38:27.904510 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:38:27.904632 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14418,"mutex_wait_us":7,"thread_start_us":15208,"threads_started":1}
I20260929 16:38:27.905679 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416fbebf92f4ab9960d32735c50f819" is_pre_election: true
I20260929 16:38:27.905952 581 raft_consensus.cc:2379] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d79edb985474f7a85a6ee63d4a20f07 in current term 1: Already voted for candidate 6a686bd259d84bf0b893bbdbb90252c8 in this term.
I20260929 16:38:27.906288 450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146d65421c348cd9430806668dad8a1" candidate_uuid: "2d79edb985474f7a85a6ee63d4a20f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a686bd259d84bf0b893bbdbb90252c8" is_pre_election: true
I20260929 16:38:27.907466 32721 leader_election.cc:302] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d79edb985474f7a85a6ee63d4a20f07; no voters: 6a686bd259d84bf0b893bbdbb90252c8, e416fbebf92f4ab9960d32735c50f819
I20260929 16:38:27.908105 650 raft_consensus.cc:2726] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:38:27.909319 32645 catalog_manager.cc:5495] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2). New cstate: current_term: 1 leader_uuid: "6a686bd259d84bf0b893bbdbb90252c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a686bd259d84bf0b893bbdbb90252c8" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 } health_report { overall_health: UNKNOWN } } }
I20260929 16:38:27.934767 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.929529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.17.205.2:41517)
0929 16:38:27.929723 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.929728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.929749 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:38:27.929939 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.929943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.930031 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.930151 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:38:27.930463 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.931255 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.932785 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.933221 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.933311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.933727 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:38:27.934223 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:38:27.934332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.934421 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.934485 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:38:27.943136 581 raft_consensus.cc:1261] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Refusing update from remote peer 6a686bd259d84bf0b893bbdbb90252c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:27.943131 317 raft_consensus.cc:1261] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Refusing update from remote peer 6a686bd259d84bf0b893bbdbb90252c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:27.943981 656 consensus_queue.cc:1035] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d79edb985474f7a85a6ee63d4a20f07" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:27.944336 649 consensus_queue.cc:1035] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e416fbebf92f4ab9960d32735c50f819" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 37983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:27.956984 660 mvcc.cc:205] Tried to move back new op lower bound from 7334706822924255232 to 7334706822780231680. Current Snapshot: MvccSnapshot[applied={T|T < 7334706822924255232}]
I20260929 16:38:27.970254 32645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:27.970535 32645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:27.974318 32645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Sending DeleteTablet for 3 replicas of tablet 7146d65421c348cd9430806668dad8a1
I20260929 16:38:27.975642 32766 tablet_service.cc:1514] Processing DeleteTablet for tablet 7146d65421c348cd9430806668dad8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:27 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:48920
I20260929 16:38:27.975638 561 tablet_service.cc:1514] Processing DeleteTablet for tablet 7146d65421c348cd9430806668dad8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:27 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:50524
I20260929 16:38:27.975656 430 tablet_service.cc:1514] Processing DeleteTablet for tablet 7146d65421c348cd9430806668dad8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:27 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:32990
I20260929 16:38:27.976831 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32683
I20260929 16:38:27.988004 669 tablet_replica.cc:330] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: stopping tablet replica
I20260929 16:38:27.988513 669 raft_consensus.cc:2229] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:27.989084 669 raft_consensus.cc:2258] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:27.989938 668 tablet_replica.cc:330] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: stopping tablet replica
I20260929 16:38:27.990381 668 raft_consensus.cc:2229] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:27.990883 668 raft_consensus.cc:2258] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:27.992489 670 tablet_replica.cc:330] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: stopping tablet replica
I20260929 16:38:27.992825 668 ts_tablet_manager.cc:1905] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:27.992914 670 raft_consensus.cc:2229] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:38:27.993683 669 ts_tablet_manager.cc:1905] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:27.993950 670 raft_consensus.cc:2258] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:27.996457 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:27.977990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.17.205.1:36603)
0929 16:38:27.978164 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:27.978171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:27.978191 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:38:27.978255 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:27.978261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:27.978344 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:27.978464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:27.980188 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.981396 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.983205 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:27.983929 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:27.984051 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:27.987509 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:27.987527 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:27.987548 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:27.987586 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:27.991827 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.993506 (+ 1679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.993514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.993518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.993678 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:27.994258 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:27.994278 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:27.994282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:27.994774 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:27.995059 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:27.995848 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 16:38:27.996075 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:38:27.998842 358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:28.001647 669 ts_tablet_manager.cc:1918] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:28.001827 669 log.cc:1196] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/7146d65421c348cd9430806668dad8a1
I20260929 16:38:28.002377 669 ts_tablet_manager.cc:1939] T 7146d65421c348cd9430806668dad8a1 P e416fbebf92f4ab9960d32735c50f819: Deleting consensus metadata
I20260929 16:38:28.004489 32630 catalog_manager.cc:4841] TS e416fbebf92f4ab9960d32735c50f819 (127.17.205.3:37983): tablet 7146d65421c348cd9430806668dad8a1 (table test-table [id=3323d785a3d8467f938c4c1123451cba]) successfully deleted
I20260929 16:38:28.005712 670 ts_tablet_manager.cc:1905] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:28.012707 670 ts_tablet_manager.cc:1918] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:28.012885 670 log.cc:1196] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/7146d65421c348cd9430806668dad8a1
I20260929 16:38:28.013373 670 ts_tablet_manager.cc:1939] T 7146d65421c348cd9430806668dad8a1 P 6a686bd259d84bf0b893bbdbb90252c8: Deleting consensus metadata
I20260929 16:38:28.014941 32632 catalog_manager.cc:4841] TS 6a686bd259d84bf0b893bbdbb90252c8 (127.17.205.2:41517): tablet 7146d65421c348cd9430806668dad8a1 (table test-table [id=3323d785a3d8467f938c4c1123451cba]) successfully deleted
I20260929 16:38:28.116946 668 ts_tablet_manager.cc:1918] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:28.117211 668 log.cc:1196] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/7146d65421c348cd9430806668dad8a1
I20260929 16:38:28.117653 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32683
I20260929 16:38:28.138921 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 367
W20260929 16:38:28.139117 32632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:36603 (error 108)
I20260929 16:38:28.141414 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.140940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48928)
0929 16:38:28.141128 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.141242 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:38:28.141767 32632 catalog_manager.cc:4584] TS 2d79edb985474f7a85a6ee63d4a20f07 (127.17.205.1:36603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7146d65421c348cd9430806668dad8a1: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
I20260929 16:38:28.155453 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.139832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.17.205.2:41517)
0929 16:38:28.140074 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:28.140081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:28.140107 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:38:28.140263 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:28.140269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:28.140352 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:28.140440 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:28.142277 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:28.143396 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:28.145373 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:28.145861 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:28.145967 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:28.149239 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:28.149254 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:28.149271 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:28.149312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:28.152666 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:28.153506 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:28.153514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:28.153516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:28.153652 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:28.154040 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:28.154044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:28.154046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:28.154435 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:28.154703 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:28.154973 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:38:28.155144 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 16:38:28.156086 490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:28.195297 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.194738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48936)
0929 16:38:28.194985 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.195100 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:38:28.196367 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.195930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48948)
0929 16:38:28.196097 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.196183 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:38:28.256891 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.256346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48950)
0929 16:38:28.256578 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.256696 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:38:28.257915 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.257513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48954)
0929 16:38:28.257651 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.257738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 16:38:28.269579 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 367
I20260929 16:38:28.285401 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 498
I20260929 16:38:28.303124 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.286122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.17.205.3:37983)
0929 16:38:28.286297 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:28.286301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:28.286321 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:38:28.286476 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:28.286480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:28.286554 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:28.286634 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:28.287936 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:28.288592 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:28.289790 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:28.290232 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:28.290325 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:28.297284 (+ 6959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:28.297301 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:28.297318 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:28.297361 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:28.300341 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:28.301094 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:28.301101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:28.301104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:28.301233 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:28.301594 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:28.301598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:28.301600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:28.301995 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:28.302266 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:28.302557 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:38:28.302784 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:38:28.303747 621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:28.338533 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.337994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48966)
0929 16:38:28.338221 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.338338 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:38:28.339687 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.339200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48968)
0929 16:38:28.339434 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.339544 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 16:38:28.401371 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 498
I20260929 16:38:28.415556 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 32613
I20260929 16:38:28.416765 32676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:28.529948 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:28.529522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36603 (local address 127.0.0.1:48978)
0929 16:38:28.529708 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:28.529804 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:38:28.530005 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 32613
I20260929 16:38:28.548372 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--webserver_interface=127.17.205.62
--webserver_port=44997
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:28.880257 676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:28.880554 676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:28.880671 676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:28.888064 676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:28.888188 676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:28.888236 676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:28.888312 676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:28.888372 676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:28.898831 676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:37887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:28.900269 676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:28.901660 676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:28.913414 681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:28.913439 682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:28.913416 684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:28.914266 676 server_base.cc:1034] running on GCE node
I20260929 16:38:28.915108 676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:28.916649 676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:28.917858 676 hybrid_clock.cc:648] HybridClock initialized: now 1790699908917790 us; error 67 us; skew 500 ppm
I20260929 16:38:28.918210 676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:28 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699908, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:28.921351 676 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:28.922905 676 webserver.cc:460] Webserver started at http://127.17.205.62:44997/ using document root <none> and password file <none>
I20260929 16:38:28.923663 676 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:28.923784 676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:28.929225 676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:38:28.932201 691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:28.933594 676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:38:28.933765 676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "626264d0a1be65442480cd118bd17a71"
server_key_iv: "38badab909a32b25ae693823995ad6f4"
server_key_version: "encryptionkey@0"
I20260929 16:38:28.934768 676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:28.959105 676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:28.963622 676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:28.963874 676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:28.981297 676 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:37887
I20260929 16:38:28.981318 742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:37887 every 8 connection(s)
I20260929 16:38:28.982825 676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:28.989892 743 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:38:28.990372 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 676
I20260929 16:38:28.991328 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:36603
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=44173
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:28.993541 743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap starting.
I20260929 16:38:29.019594 743 log.cc:824] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:29.040889 743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:38:29.041575 743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1: Bootstrap complete.
I20260929 16:38:29.055130 743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:29.056603 743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Initialized, Role: FOLLOWER
I20260929 16:38:29.057492 743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:29.057806 743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:29.057955 743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:29.058163 743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:38:29.064131 743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:29.064775 743 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef1e69903bc4e51b10e5ac2f055c9a1; no voters:
I20260929 16:38:29.065539 743 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:38:29.065689 747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:38:29.066669 747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [term 3 LEADER]: Becoming Leader. State: Replica: 6ef1e69903bc4e51b10e5ac2f055c9a1, State: Running, Role: LEADER
I20260929 16:38:29.067164 747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } }
I20260929 16:38:29.067777 743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:29.069459 748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:29.069818 748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:29.070472 749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef1e69903bc4e51b10e5ac2f055c9a1. Latest consensus state: current_term: 3 leader_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef1e69903bc4e51b10e5ac2f055c9a1" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37887 } } }
I20260929 16:38:29.071012 749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef1e69903bc4e51b10e5ac2f055c9a1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:29.073714 752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:29.082230 752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:29.083774 752 catalog_manager.cc:1259] Loaded cluster ID: 5aea9e5196ca41b99f505a5cff91495f
I20260929 16:38:29.083937 752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:38:29.085943 752 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feffccf29ad google::LogMessage::Fail() at ??:0
@ 0x7feffccf588a google::LogMessage::SendToLog() at ??:0
@ 0x7feffccf24de google::LogMessage::Flush() at ??:0
@ 0x7feffccf3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff012725512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff012722976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0127aa610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff0127aa439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fefff93e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fefff94191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fefff941799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fefff91ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff002bf66db start_thread at ??:0
@ 0x7feffbb8c61f clone at ??:0
W20260929 16:38:29.391590 745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:29.391880 745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:29.392015 745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:29.399320 745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:29.399437 745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:29.399605 745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:29.409216 745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:36603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=44173
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:29.410459 745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:29.411791 745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:29.424029 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:29.424110 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:29.424146 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:29.424642 745 server_base.cc:1034] running on GCE node
I20260929 16:38:29.425241 745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:29.426198 745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:29.427380 745 hybrid_clock.cc:648] HybridClock initialized: now 1790699909427328 us; error 43 us; skew 500 ppm
I20260929 16:38:29.427682 745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:29 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699909, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:29.431525 745 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:29.433403 745 webserver.cc:460] Webserver started at http://127.17.205.1:44173/ using document root <none> and password file <none>
I20260929 16:38:29.434309 745 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:29.434449 745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:29.440384 745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:38:29.443105 794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:29.444475 745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:38:29.444617 745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "2d79edb985474f7a85a6ee63d4a20f07"
format_stamp: "Formatted at 2026-09-29 16:38:23 on dist-test-slave-3h01"
server_key: "8df36b619ced95e5869116dbda0966d9"
server_key_iv: "b2dabef4b6a16de2120f02a139c3e77e"
server_key_version: "encryptionkey@0"
I20260929 16:38:29.445597 745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:29.458475 745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:29.462359 745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:29.462607 745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:29.463631 745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:29.465358 801 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 16:38:29.470775 801 ts_tablet_manager.cc:1852] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 16:38:29.471817 801 ts_tablet_manager.cc:1905] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:29.479161 801 ts_tablet_manager.cc:1918] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:29.479470 801 ts_tablet_manager.cc:1939] T 7146d65421c348cd9430806668dad8a1 P 2d79edb985474f7a85a6ee63d4a20f07: Deleting consensus metadata
I20260929 16:38:29.481587 745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 16:38:29.481694 745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.001s
I20260929 16:38:29.481860 745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:29.481940 745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:29.512288 745 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:36603
I20260929 16:38:29.512343 907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:36603 every 8 connection(s)
I20260929 16:38:29.513604 745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:29.518409 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 745
I20260929 16:38:29.518630 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.516558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:35717)
0929 16:38:29.517276 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.517512 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":271,"threads_started":1}
I20260929 16:38:29.519114 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:41517
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=43351
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:29.519244 908 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:37887 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:37887: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
I20260929 16:38:29.520107 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.519705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:59287)
0929 16:38:29.519880 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.519975 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:38:29.520962 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.520650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:37881)
0929 16:38:29.520772 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.520846 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:38:29.521198 908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:38:29.839916 911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:29.840205 911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:29.840327 911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:29.847640 911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:29.847756 911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:29.847904 911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:29.859025 911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=43351
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:29.860401 911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:29.861783 911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:29.875049 917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:29.875116 916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:29.875092 919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:29.875840 911 server_base.cc:1034] running on GCE node
I20260929 16:38:29.876447 911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:29.877429 911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:29.878628 911 hybrid_clock.cc:648] HybridClock initialized: now 1790699909878568 us; error 61 us; skew 500 ppm
I20260929 16:38:29.878931 911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:29 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699909, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:29.881970 911 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:29.883517 911 webserver.cc:460] Webserver started at http://127.17.205.2:43351/ using document root <none> and password file <none>
I20260929 16:38:29.884189 911 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:29.884312 911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:29.889881 911 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:38:29.892705 926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:29.894080 911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:38:29.894232 911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "6a686bd259d84bf0b893bbdbb90252c8"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "5182ec913ec40404f73cffa8f8613618"
server_key_iv: "d0ba0f9df92beea2093180ae1330616f"
server_key_version: "encryptionkey@0"
I20260929 16:38:29.895254 911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:29.911087 911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:29.915405 911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:29.915659 911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:29.916625 911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:29.917922 911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:29.918011 911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:29.918118 911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:29.918186 911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:29.947846 911 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:41517
I20260929 16:38:29.947911 1038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:41517 every 8 connection(s)
I20260929 16:38:29.949329 911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:29.949733 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 911
I20260929 16:38:29.950467 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:37983
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:37887
--webserver_port=36285
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:38:29.954792 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.952520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:54695)
0929 16:38:29.953333 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.953605 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
W20260929 16:38:29.955569 1039 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:37887 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:37887: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
I20260929 16:38:29.956507 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.956134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:49295)
0929 16:38:29.956290 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.956369 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:38:29.957383 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:29.957052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:60765)
0929 16:38:29.957178 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:29.957261 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:38:29.957608 1039 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:38:30.270546 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:30.270819 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:30.270961 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:30.278180 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:30.278288 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:30.278437 1042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:30.288074 1042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:37983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=36285
--tserver_master_addrs=127.17.205.62:37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:30.289424 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:30.290776 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:30.303695 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:30.303755 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:30.303733 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:30.304376 1042 server_base.cc:1034] running on GCE node
I20260929 16:38:30.304978 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:30.305949 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:30.307147 1042 hybrid_clock.cc:648] HybridClock initialized: now 1790699910307099 us; error 37 us; skew 500 ppm
I20260929 16:38:30.307476 1042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:30 dist-test-slave-3h01 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699910, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:30.310505 1042 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:30.311930 1042 webserver.cc:460] Webserver started at http://127.17.205.3:36285/ using document root <none> and password file <none>
I20260929 16:38:30.312618 1042 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:30.312717 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:30.318051 1042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:38:30.320791 1057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:30.322184 1042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:38:30.322353 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "e416fbebf92f4ab9960d32735c50f819"
format_stamp: "Formatted at 2026-09-29 16:38:24 on dist-test-slave-3h01"
server_key: "de97d47c1255267836f06cc3fd46e0d9"
server_key_iv: "e5d671293d89dd26f14c52c8a126fc26"
server_key_version: "encryptionkey@0"
I20260929 16:38:30.323343 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:30.340265 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:30.344446 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:30.344676 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:30.345590 1042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:30.346869 1042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:30.346973 1042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:30.347055 1042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:30.347106 1042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:30.376543 1042 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:37983
I20260929 16:38:30.376607 1169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:37983 every 8 connection(s)
I20260929 16:38:30.377951 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:30.382710 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:30.380949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:42511)
0929 16:38:30.381652 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:30.381856 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":255,"threads_started":1}
W20260929 16:38:30.383409 1170 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:37887 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:37887: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
I20260929 16:38:30.383595 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1042
I20260929 16:38:30.384250 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:30.383923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:39107)
0929 16:38:30.384055 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:30.384122 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 16:38:30.385139 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:30.384830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:33033)
0929 16:38:30.384968 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:30.385038 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:38:30.385380 1170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:38:30.522483 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:30.521742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:50329)
0929 16:38:30.522170 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:30.522272 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":199,"threads_started":1}
I20260929 16:38:30.959110 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:30.958232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:52889)
0929 16:38:30.958726 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:30.958849 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260929 16:38:31.386842 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:31.385958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:51993)
0929 16:38:31.386406 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:31.386557 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":201,"threads_started":1}
I20260929 16:38:31.524097 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:31.523385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:41035)
0929 16:38:31.523811 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:31.523898 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
I20260929 16:38:31.960956 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:31.960064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:46325)
0929 16:38:31.960613 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:31.960732 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":253,"threads_started":1}
I20260929 16:38:32.388489 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:32.387770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:53905)
0929 16:38:32.388237 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:32.388324 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260929 16:38:32.525925 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:32.525117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:60495)
0929 16:38:32.525632 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:32.525743 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
I20260929 16:38:32.962677 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:32.961920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:38939)
0929 16:38:32.962381 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:32.962469 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
I20260929 16:38:33.390151 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:33.389401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:56917)
0929 16:38:33.389896 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:33.389984 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
I20260929 16:38:33.527683 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:33.526847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:39697)
0929 16:38:33.527359 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:33.527493 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":242,"threads_started":1}
I20260929 16:38:33.964414 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:33.963611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:45577)
0929 16:38:33.964104 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:33.964205 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":227,"threads_started":1}
I20260929 16:38:34.392107 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:34.391121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:60605)
0929 16:38:34.391821 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:34.391902 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":369,"threads_started":1}
I20260929 16:38:34.529284 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:34.528569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:42469)
0929 16:38:34.529028 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:34.529116 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260929 16:38:34.966087 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:34.965331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:57045)
0929 16:38:34.965801 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:34.965891 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260929 16:38:35.393793 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:35.393073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:58341)
0929 16:38:35.393531 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:35.393624 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260929 16:38:35.531060 1189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:35.530300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:55977)
0929 16:38:35.530782 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:35.530875 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260929 16:38:35.967764 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:35.966975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:45547)
0929 16:38:35.967483 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:35.967585 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260929 16:38:36.395581 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:36.394750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:54879)
0929 16:38:36.395306 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:36.395410 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260929 16:38:36.532864 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:36.532031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:52703)
0929 16:38:36.532566 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:36.532666 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":240,"threads_started":1}
I20260929 16:38:36.969677 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:36.968795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:34309)
0929 16:38:36.969325 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:36.969421 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":232,"threads_started":1}
I20260929 16:38:37.397228 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:37.396480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:45007)
0929 16:38:37.396939 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:37.397021 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":191,"threads_started":1}
I20260929 16:38:37.534505 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:37.533795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:57581)
0929 16:38:37.534237 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:37.534321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260929 16:38:37.971406 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:37.970640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:44559)
0929 16:38:37.971099 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:37.971191 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":230,"threads_started":1}
I20260929 16:38:38.398775 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:38.398070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:47955)
0929 16:38:38.398525 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:38.398608 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260929 16:38:38.536234 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:38.535428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:58957)
0929 16:38:38.535948 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:38.536070 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260929 16:38:38.973299 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:38.972469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:51645)
0929 16:38:38.973014 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:38.973115 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":249,"threads_started":1}
I20260929 16:38:39.400454 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:39.399683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:38775)
0929 16:38:39.400192 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:39.400280 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":242,"threads_started":1}
I20260929 16:38:39.538010 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:39.537224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:54417)
0929 16:38:39.537729 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:39.537822 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
I20260929 16:38:39.975171 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:39.974373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:37243)
0929 16:38:39.974835 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:39.974968 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260929 16:38:40.402112 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:40.401356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:43943)
0929 16:38:40.401851 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:40.401941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
I20260929 16:38:40.539652 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:40.538869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:45427)
0929 16:38:40.539368 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:40.539455 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260929 16:38:40.976874 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:40.976158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:35017)
0929 16:38:40.976603 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:40.976700 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260929 16:38:41.403910 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:41.403103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:38211)
0929 16:38:41.403602 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:41.403718 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260929 16:38:41.541272 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:41.540553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:44157)
0929 16:38:41.541006 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:41.541097 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260929 16:38:41.978464 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:41.977791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:50131)
0929 16:38:41.978225 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:41.978310 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":209,"threads_started":1}
I20260929 16:38:42.405676 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:42.404955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:43827)
0929 16:38:42.405418 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:42.405518 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260929 16:38:42.542886 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:42.542145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:48753)
0929 16:38:42.542599 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:42.542717 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260929 16:38:42.980289 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:42.979396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:55673)
0929 16:38:42.979890 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:42.980013 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":231,"threads_started":1}
I20260929 16:38:43.407534 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:43.406691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:32821)
0929 16:38:43.407235 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:43.407342 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":235,"threads_started":1}
I20260929 16:38:43.544577 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:43.543824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:47117)
0929 16:38:43.544271 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:43.544371 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260929 16:38:43.981987 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:43.981251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:52479)
0929 16:38:43.981709 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:43.981809 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260929 16:38:44.409344 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:44.408547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:59429)
0929 16:38:44.409035 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:44.409154 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260929 16:38:44.546550 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:44.545776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.1:34665)
0929 16:38:44.546282 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:44.546379 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260929 16:38:44.984115 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:44.983329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.2:49753)
0929 16:38:44.983795 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:44.983894 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260929 16:38:45.390046 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 745
I20260929 16:38:45.408598 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:45.391177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.17.205.1:36603)
0929 16:38:45.391838 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:45.391857 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:45.391904 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:38:45.394231 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:45.394244 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:45.394712 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:45.395015 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:45.396491 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.397794 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.399139 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.399980 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.400155 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:45.402834 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:45.402842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:45.402860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:45.402901 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:45.405614 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.406252 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.406259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.406266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.406430 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.406722 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.406724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.406726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.407313 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:45.407534 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:45.407866 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:38:45.408299 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
I20260929 16:38:45.410063 903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:45.411237 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:45.410453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37887 (local address 127.17.205.3:49907)
0929 16:38:45.410911 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:45.411000 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
I20260929 16:38:45.499310 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 745
I20260929 16:38:45.513899 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 911
I20260929 16:38:45.532532 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:45.514906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.17.205.2:41517)
0929 16:38:45.515485 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:45.515499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:45.515537 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:38:45.517151 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:45.517162 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:45.517638 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:45.518005 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:45.519644 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.520960 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.522280 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.523049 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.523281 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:45.525888 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:45.525897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:45.525914 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:45.525964 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:45.529060 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.529771 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.529779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.529787 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.529977 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.530370 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.530374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.530376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.530981 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:45.531318 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:45.531795 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 16:38:45.532244 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":302,"threads_started":1}
I20260929 16:38:45.534122 1034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:45.619802 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 911
I20260929 16:38:45.635465 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1042
I20260929 16:38:45.654176 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:45.636551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.17.205.3:37983)
0929 16:38:45.637189 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:45.637204 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:45.637249 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:38:45.638896 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:45.638906 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:45.639435 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:45.639817 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:45.641378 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.642628 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.643970 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:45.644691 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:45.644872 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:45.647632 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:45.647640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:45.647658 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:45.647701 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:45.650590 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.651395 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.651403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.651411 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.651619 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:45.652031 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:45.652035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:45.652037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:45.652627 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:45.652951 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:45.653404 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 16:38:45.653871 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":282,"threads_started":1}
I20260929 16:38:45.655721 1165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:45.745530 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1042
2026-09-29T16:38:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24091 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:38:45 dist-test-slave-3h01 krb5kdc[1227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:38:45 dist-test-slave-3h01 krb5kdc[1227](info): set up 2 sockets
Sep 29 16:38:45 dist-test-slave-3h01 krb5kdc[1227](info): commencing operation
krb5kdc: starting...
W20260929 16:38:46.767622 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:38:46 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:38:46.993021 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:47.319856 1240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:47.320184 1240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:47.320335 1240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:47.327526 1240 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:38:47.327628 1240 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:38:47.327692 1240 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:38:47.327761 1240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:38:47.327839 1240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:47.327939 1240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:47.327999 1240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:38:47.328073 1240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:38:47.338428 1240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:44929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:47.339818 1240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:47.341219 1240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:47.353333 1248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:47.353385 1246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:47.353421 1245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:47.353701 1240 server_base.cc:1034] running on GCE node
I20260929 16:38:47.354748 1240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:47.356254 1240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:47.357473 1240 hybrid_clock.cc:648] HybridClock initialized: now 1790699927357415 us; error 53 us; skew 500 ppm
I20260929 16:38:47.357805 1240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:47 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699927, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:47.360924 1240 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:38:47.362352 1240 webserver.cc:460] Webserver started at http://127.17.205.62:35041/ using document root <none> and password file <none>
I20260929 16:38:47.363059 1240 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:47.363144 1240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:47.363456 1240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:47.366528 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "ecea2d18dba1416995e0ac979bc12425"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "e36e5741553d06ccb6bc7a6ab67393ee"
server_key_iv: "5674c5d063bfe757eabc5230da5a8519"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.367311 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "ecea2d18dba1416995e0ac979bc12425"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "e36e5741553d06ccb6bc7a6ab67393ee"
server_key_iv: "5674c5d063bfe757eabc5230da5a8519"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.372537 1240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:38:47.376220 1255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:47.377614 1240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:38:47.377756 1240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "ecea2d18dba1416995e0ac979bc12425"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "e36e5741553d06ccb6bc7a6ab67393ee"
server_key_iv: "5674c5d063bfe757eabc5230da5a8519"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.377939 1240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:47.393378 1240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:47.397578 1240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:47.397847 1240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:47.414063 1240 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:44929
I20260929 16:38:47.414043 1306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:44929 every 8 connection(s)
I20260929 16:38:47.415522 1240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:38:47.419453 1307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:47.425081 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1240
I20260929 16:38:47.425300 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:38:47.426167 18228 external_mini_cluster.cc:1428] Setting key c9447d6b7f172ce69c9650409c59b9c4
I20260929 16:38:47.431994 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap starting.
I20260929 16:38:47.435855 1307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:47.437048 1307 log.cc:824] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:47.440021 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: No bootstrap required, opened a new log
Sep 29 16:38:47 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:47.448308 1307 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:38:47.448710 1307 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:47.448800 1307 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Initialized, Role: FOLLOWER
I20260929 16:38:47.449540 1307 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:38:47.449828 1307 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:38:47.449962 1307 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:38:47.450143 1307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:47.451134 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:47.428336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.17.205.62:44929)
0929 16:38:47.429198 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:47.429215 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:47.429273 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:38:47.431659 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:47.431692 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:47.432250 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:47.432731 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:47.434114 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:47.436088 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:47.437610 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:47.438416 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:47.438591 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:47.442815 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:47.442827 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:47.442844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:47.442891 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:47.446321 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:47.447068 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:47.447082 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:47.447091 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:47.447333 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:47.447732 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:47.447737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:47.447741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:47.448346 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:47.448614 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:47.449149 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 16:38:47.449621 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":358,"threads_started":1}
I20260929 16:38:47.453064 1307 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:38:47.453657 1307 leader_election.cc:302] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecea2d18dba1416995e0ac979bc12425; no voters:
I20260929 16:38:47.454330 1307 leader_election.cc:288] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:38:47.454456 1312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:47.455133 1312 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 LEADER]: Becoming Leader. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Running, Role: LEADER
I20260929 16:38:47.455698 1312 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:38:47.456444 1307 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:38:47.458298 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecea2d18dba1416995e0ac979bc12425. Latest consensus state: current_term: 1 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:38:47.458779 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:47.459122 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:38:47.459519 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:38:47.461055 1321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:38:47.466147 1321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:38:47.474886 1321 catalog_manager.cc:1347] Generated new cluster ID: 2bb8417b976248fd96ac4e6fd36bb51f
I20260929 16:38:47.475018 1321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:38:47.495859 1321 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:38:47.497512 1321 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:38:47.506173 1321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Generated new TSK 0
I20260929 16:38:47.507066 1321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:38:47.558351 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:47.884297 1333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:47.884598 1333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:47.884716 1333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:47.892143 1333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:47.892261 1333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:47.892405 1333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:38:47.902534 1333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:47.904004 1333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:47.905370 1333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:47.918964 1338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:47.918898 1341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:47.918921 1339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:47.924628 1333 server_base.cc:1034] running on GCE node
I20260929 16:38:47.925290 1333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:47.926237 1333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:47.927433 1333 hybrid_clock.cc:648] HybridClock initialized: now 1790699927927380 us; error 46 us; skew 500 ppm
I20260929 16:38:47.927738 1333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:47 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699927, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:47.934688 1333 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:38:47.936214 1333 webserver.cc:460] Webserver started at http://127.17.205.1:33133/ using document root <none> and password file <none>
I20260929 16:38:47.936893 1333 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:47.936990 1333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:47.937254 1333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:47.940227 1333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "1e2ed7d0c19b4095be8e4662f72f20ee"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "cc7238d42d70d1177f631a636d121d6a"
server_key_iv: "9d26a965c7fa9c13822d417fcc2a3737"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.940893 1333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "1e2ed7d0c19b4095be8e4662f72f20ee"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "cc7238d42d70d1177f631a636d121d6a"
server_key_iv: "9d26a965c7fa9c13822d417fcc2a3737"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.945904 1333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:38:47.949287 1348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:47.950649 1333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 16:38:47.950784 1333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "1e2ed7d0c19b4095be8e4662f72f20ee"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "cc7238d42d70d1177f631a636d121d6a"
server_key_iv: "9d26a965c7fa9c13822d417fcc2a3737"
server_key_version: "encryptionkey@0"
I20260929 16:38:47.950951 1333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:47.962703 1333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:47.966938 1333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:47.967250 1333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:47.968202 1333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:47.969506 1333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:47.969578 1333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:47.969694 1333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:47.969743 1333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.000335 1333 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42513
I20260929 16:38:48.000380 1460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42513 every 8 connection(s)
I20260929 16:38:48.001614 1333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:38:48.010206 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1333
I20260929 16:38:48.010401 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:38:48.011032 18228 external_mini_cluster.cc:1428] Setting key e65812fe075afb3d5549304947383740
Sep 29 16:38:48 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699927, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:48.021680 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:48.004166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:45949 (local address 127.17.205.62:44929)
0929 16:38:48.004590 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:48.004595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:48.005579 (+ 984us) server_negotiation.cc:407] Connection header received
0929 16:38:48.007482 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:48.007488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:48.007571 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:48.007667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:48.009209 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.010059 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.011760 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.012258 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.012380 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:48.016167 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:48.016172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:48.016175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:48.016207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:48.018611 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:48.019443 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:48.019447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:48.019450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:48.019584 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:48.019996 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:48.020000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:48.020001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:48.020258 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:48.020402 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:48.021084 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 16:38:48.021321 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260929 16:38:48.022464 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:48.004715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:45949)
0929 16:38:48.005427 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:48.005466 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:48.007171 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:48.007915 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:48.007924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:48.008455 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:48.009038 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:48.009057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.010216 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.010220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:48.011614 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:48.011632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.012451 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.012456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:48.012678 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:48.013310 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:48.013341 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:48.015954 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:48.018741 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:48.018750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:48.018769 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:48.019343 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:48.019669 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:48.019673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:48.019676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:48.019909 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:48.020377 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:48.020384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:48.020728 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 16:38:48.020955 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:38:48.021357 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":297,"threads_started":1}
I20260929 16:38:48.023990 1461 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:38:48.024286 1461 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:48.024964 1461 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:38:48.026950 1272 ts_manager.cc:194] Registered new tserver with Master: 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1:42513)
I20260929 16:38:48.028455 1272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:45949
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:38:48.062076 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:48.378485 1467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:48.378794 1467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:48.378961 1467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:48.386469 1467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:48.386600 1467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:48.386765 1467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:38:48.396694 1467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:48.398030 1467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:48.399415 1467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:48.412545 1472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:48.412482 1473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:48.412511 1475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:48.413672 1467 server_base.cc:1034] running on GCE node
I20260929 16:38:48.414301 1467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:48.415338 1467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:48.416563 1467 hybrid_clock.cc:648] HybridClock initialized: now 1790699928416500 us; error 53 us; skew 500 ppm
I20260929 16:38:48.416877 1467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:48 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:48.420047 1467 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:38:48.421617 1467 webserver.cc:460] Webserver started at http://127.17.205.2:38369/ using document root <none> and password file <none>
I20260929 16:38:48.422318 1467 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:48.422432 1467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:48.422745 1467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:48.425731 1467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "bda3126a74904629a6987ecb46cf86f0"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "de92540bb80867570b6edeefbb336fe2"
server_key_iv: "bf2920f346ba6cdc5797772e77e07709"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.426499 1467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "bda3126a74904629a6987ecb46cf86f0"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "de92540bb80867570b6edeefbb336fe2"
server_key_iv: "bf2920f346ba6cdc5797772e77e07709"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.431669 1467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:38:48.435113 1482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.436492 1467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:38:48.436645 1467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "bda3126a74904629a6987ecb46cf86f0"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "de92540bb80867570b6edeefbb336fe2"
server_key_iv: "bf2920f346ba6cdc5797772e77e07709"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.436822 1467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:48.469460 1467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:48.473721 1467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:48.473997 1467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:48.475024 1467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:48.476420 1467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:48.476493 1467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.476603 1467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:48.476653 1467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.506768 1467 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:43299
I20260929 16:38:48.506790 1594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:43299 every 8 connection(s)
I20260929 16:38:48.508128 1467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:38:48.513325 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1467
I20260929 16:38:48.513535 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:38:48.514098 18228 external_mini_cluster.cc:1428] Setting key f4b87e2192224d7d2144f4c5911945c8
Sep 29 16:38:48 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:48.529587 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:48.510627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:38439 (local address 127.17.205.62:44929)
0929 16:38:48.510824 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:48.510830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:48.512006 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 16:38:48.514291 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:48.514296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:48.514372 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:48.514457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:48.516301 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.517007 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.519188 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.519644 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.519736 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:48.523917 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:48.523922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:48.523926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:48.523964 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:48.526438 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:48.527257 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:48.527262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:48.527264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:48.527397 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:48.527820 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:48.527824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:48.527826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:48.528107 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:48.528243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:48.528976 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 16:38:48.529237 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:38:48.530606 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:48.511145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:38439)
0929 16:38:48.511851 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:48.511890 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:48.513936 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:48.514761 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:48.514771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:48.515438 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:48.516130 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:48.516156 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.517172 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.517178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:48.519062 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:48.519077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:48.519861 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:48.519867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:48.520141 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:48.520841 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:48.520879 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:48.523665 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:48.526599 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:48.526609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:48.526628 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:48.527082 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:48.527503 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:48.527509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:48.527513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:48.527703 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:48.528227 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:48.528235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:48.528603 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 16:38:48.528861 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 16:38:48.529431 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
I20260929 16:38:48.532188 1595 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:38:48.532480 1595 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:48.533213 1595 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:38:48.534662 1272 ts_manager.cc:194] Registered new tserver with Master: bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:38:48.535483 1272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:38439
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:38:48.564322 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:38:48.888952 1600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:38:48.889235 1600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:38:48.889333 1600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:38:48.896782 1600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:38:48.896898 1600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:38:48.897039 1600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:38:48.907063 1600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:38:48.908353 1600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:38:48.909648 1600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:38:48.922791 1605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:48.922859 1606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:38:48.922829 1608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:38:48.923656 1600 server_base.cc:1034] running on GCE node
I20260929 16:38:48.924211 1600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:38:48.925060 1600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:38:48.926245 1600 hybrid_clock.cc:648] HybridClock initialized: now 1790699928926190 us; error 50 us; skew 500 ppm
I20260929 16:38:48.926540 1600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:38:48 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:38:48.929490 1600 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:38:48.930826 1600 webserver.cc:460] Webserver started at http://127.17.205.3:42393/ using document root <none> and password file <none>
I20260929 16:38:48.931576 1600 fs_manager.cc:359] Metadata directory not provided
I20260929 16:38:48.931690 1600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:38:48.931967 1600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:38:48.934768 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "a7041ce305364d89971a8939bbcc2446"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "b4c35c16e094bf4bf735ec12de1737cd"
server_key_iv: "13c3da1fe6a939940aa97f683eeb09a7"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.935542 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "a7041ce305364d89971a8939bbcc2446"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "b4c35c16e094bf4bf735ec12de1737cd"
server_key_iv: "13c3da1fe6a939940aa97f683eeb09a7"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.940490 1600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:38:48.943845 1615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.945116 1600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:38:48.945259 1600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "a7041ce305364d89971a8939bbcc2446"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "b4c35c16e094bf4bf735ec12de1737cd"
server_key_iv: "13c3da1fe6a939940aa97f683eeb09a7"
server_key_version: "encryptionkey@0"
I20260929 16:38:48.945425 1600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:38:48.968974 1600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:38:48.972712 1600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:38:48.972985 1600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:38:48.973910 1600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:38:48.975183 1600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:38:48.975279 1600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:48.975396 1600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:38:48.975454 1600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:38:49.005026 1600 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:39499
I20260929 16:38:49.005097 1727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:39499 every 8 connection(s)
I20260929 16:38:49.006211 1600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:38:49.015383 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1600
I20260929 16:38:49.015575 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:38:49.016038 18228 external_mini_cluster.cc:1428] Setting key 9ee9763ccabe9561dd1fc638f43d1de7
Sep 29 16:38:49 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:38:49.024871 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.008480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:45251 (local address 127.17.205.62:44929)
0929 16:38:49.008655 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.008660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.009749 (+ 1089us) server_negotiation.cc:407] Connection header received
0929 16:38:49.011567 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.011573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.011682 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.011769 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:49.013182 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.013928 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.015668 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.016071 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.016189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.019771 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:49.019777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:49.019780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:49.019815 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:49.022270 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.022912 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.022915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.022917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.023044 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.023446 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.023450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.023452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.023719 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:49.023828 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.024452 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.024621 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:38:49.025777 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.008910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:45251)
0929 16:38:49.009589 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.009641 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.011311 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.012003 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.012011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.012490 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:38:49.013021 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.013038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.014071 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.014075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.015518 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.015533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.016438 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.016443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.016671 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.017335 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:38:49.017358 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:38:49.019595 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:38:49.022385 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:49.022391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:49.022403 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:49.022821 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:49.023137 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:38:49.023140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:38:49.023143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:38:49.023363 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:38:49.023810 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:38:49.023817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:38:49.024154 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.024370 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.024748 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":246,"threads_started":1}
I20260929 16:38:49.027046 1728 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:38:49.027357 1728 heartbeater.cc:464] Registering TS with master...
I20260929 16:38:49.028030 1728 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:38:49.029424 1272 ts_manager.cc:194] Registered new tserver with Master: a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499)
I20260929 16:38:49.030390 1272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:45251
I20260929 16:38:49.031447 1461 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:38:49.031487 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:38:49.047890 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.036525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.17.205.62:44929)
0929 16:38:49.036718 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.036724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.037089 (+ 365us) server_negotiation.cc:407] Connection header received
0929 16:38:49.037228 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.037231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.037289 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.037362 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:49.038634 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.039375 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.040648 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.041066 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.041157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.043411 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:49.043416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:49.043419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:49.043447 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:49.045700 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.046302 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.046305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.046308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.046422 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.046705 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.046707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.046709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.047002 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:49.047189 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.047464 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.047622 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:38:49.051725 1271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:38:49.058785 1271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:38:49.090703 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.071738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:43299 (local address 127.0.0.1:33282)
0929 16:38:49.081827 (+ 10089us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.081845 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.082151 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.083174 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.083181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.083233 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:49.083659 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.083674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.085581 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.085586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.088042 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.088058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.089706 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.089713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.089919 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.089974 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.090124 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.090309 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9969,"thread_start_us":8571,"threads_started":1}
I20260929 16:38:49.090909 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.072210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.17.205.2:43299)
0929 16:38:49.081418 (+ 9208us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.081427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.081941 (+ 514us) server_negotiation.cc:407] Connection header received
0929 16:38:49.082343 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.082352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.082650 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.082844 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:49.083840 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.085415 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.088220 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.089500 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.089638 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.089834 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.090198 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.090373 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9042,"thread_start_us":9050,"threads_started":1}
I20260929 16:38:49.091419 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.071272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:39499 (local address 127.0.0.1:46086)
0929 16:38:49.080718 (+ 9446us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.080765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.080945 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.083854 (+ 2909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.083859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.083882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:49.084268 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.084280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.086141 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.086146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.088895 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.088910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.090680 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.090687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.090841 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.090873 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.090942 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.091062 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9178,"thread_start_us":10298,"threads_started":1}
I20260929 16:38:49.092190 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.071303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.17.205.3:39499)
0929 16:38:49.083002 (+ 11699us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.083010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.083040 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:38:49.083129 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.083136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.083382 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.083612 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:49.084484 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.086001 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.089059 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.090360 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.090479 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.091737 (+ 1258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.091807 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.091949 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11529,"thread_start_us":11384,"threads_started":1}
I20260929 16:38:49.093660 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.083073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:34116)
0929 16:38:49.086710 (+ 3637us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.086725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.086848 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.087468 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.087477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.087501 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:49.087866 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.087874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.089460 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.089465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.091339 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.091354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.092942 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.092948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.093122 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.093153 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.093233 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.093350 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3511,"thread_start_us":3360,"threads_started":1}
I20260929 16:38:49.094663 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.082396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.17.205.1:42513)
0929 16:38:49.086387 (+ 3991us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.086395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.086976 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:38:49.087067 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.087072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.087268 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.088024 (+ 756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:49.088207 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.089314 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.091532 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.092751 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.092874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.093095 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.094181 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.094346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3827,"thread_start_us":3679,"threads_started":1}
I20260929 16:38:49.095320 1530 tablet_service.cc:1467] Processing CreateTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 (DEFAULT_TABLE table=test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:49.095715 1663 tablet_service.cc:1467] Processing CreateTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 (DEFAULT_TABLE table=test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:49.096875 1530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2728c4cf66a54d9f9043683cea4208d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:49.097014 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2728c4cf66a54d9f9043683cea4208d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:49.097533 1396 tablet_service.cc:1467] Processing CreateTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 (DEFAULT_TABLE table=test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:38:49.098591 1396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2728c4cf66a54d9f9043683cea4208d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:38:49.110634 1746 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Bootstrap starting.
I20260929 16:38:49.113417 1746 tablet_bootstrap.cc:654] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:49.114317 1748 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Bootstrap starting.
I20260929 16:38:49.114701 1746 log.cc:824] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:49.115850 1747 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Bootstrap starting.
I20260929 16:38:49.118048 1748 tablet_bootstrap.cc:654] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:49.118979 1747 tablet_bootstrap.cc:654] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Neither blocks nor log segments found. Creating new log.
I20260929 16:38:49.119318 1748 log.cc:824] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:49.120071 1747 log.cc:824] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:38:49.120985 1746 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: No bootstrap required, opened a new log
I20260929 16:38:49.121326 1746 ts_tablet_manager.cc:1397] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260929 16:38:49.122498 1748 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: No bootstrap required, opened a new log
I20260929 16:38:49.122792 1748 ts_tablet_manager.cc:1397] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:38:49.125231 1747 tablet_bootstrap.cc:492] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: No bootstrap required, opened a new log
I20260929 16:38:49.125464 1747 ts_tablet_manager.cc:1397] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 16:38:49.131846 1746 raft_consensus.cc:348] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.132067 1747 raft_consensus.cc:348] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.132340 1746 raft_consensus.cc:374] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:49.132436 1747 raft_consensus.cc:374] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:49.132475 1746 raft_consensus.cc:729] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ed7d0c19b4095be8e4662f72f20ee, State: Initialized, Role: FOLLOWER
I20260929 16:38:49.132534 1747 raft_consensus.cc:729] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda3126a74904629a6987ecb46cf86f0, State: Initialized, Role: FOLLOWER
I20260929 16:38:49.132927 1748 raft_consensus.cc:348] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.133175 1747 consensus_queue.cc:260] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.133179 1746 consensus_queue.cc:260] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.133446 1748 raft_consensus.cc:374] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:38:49.133563 1748 raft_consensus.cc:729] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7041ce305364d89971a8939bbcc2446, State: Initialized, Role: FOLLOWER
I20260929 16:38:49.134347 1748 consensus_queue.cc:260] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.135435 1595 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:38:49.136191 1747 ts_tablet_manager.cc:1428] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260929 16:38:49.137993 1746 ts_tablet_manager.cc:1428] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260929 16:38:49.138975 1728 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:38:49.140086 1748 ts_tablet_manager.cc:1428] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260929 16:38:49.202718 1752 raft_consensus.cc:482] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:38:49.203009 1752 raft_consensus.cc:504] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.204833 1752 leader_election.cc:288] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:38:49.212957 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.205488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:58911 (local address 127.17.205.2:43299)
0929 16:38:49.205699 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.205706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.206292 (+ 586us) server_negotiation.cc:407] Connection header received
0929 16:38:49.206447 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.206451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.206509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.206587 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:49.207189 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.207927 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.210644 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.211889 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.212044 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.212118 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.212569 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.212702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:38:49.212957 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.205291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:44627 (local address 127.17.205.3:39499)
0929 16:38:49.205504 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.205511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.206212 (+ 701us) server_negotiation.cc:407] Connection header received
0929 16:38:49.206407 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.206411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.206471 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.206544 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:38:49.207108 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.207921 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.210644 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.211889 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.212045 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.212124 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.212569 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.212701 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:38:49.213145 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.205259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:39499 (local address 127.17.205.1:44627)
0929 16:38:49.206078 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.206106 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.206269 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.206576 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.206580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.206603 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:49.206965 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.206977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.208077 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.208081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.210477 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.210486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.212140 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.212145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.212362 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.212438 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.212536 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.212686 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":136,"thread_start_us":418,"threads_started":1}
I20260929 16:38:49.213167 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.205289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:43299 (local address 127.17.205.1:58911)
0929 16:38:49.206178 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 16:38:49.206192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:38:49.206315 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:38:49.206613 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:38:49.206617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:38:49.206639 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:38:49.207015 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.207024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.208063 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.208068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.210466 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:38:49.210482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.212123 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.212130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:38:49.212381 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.212446 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 16:38:49.212531 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:38:49.212686 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":145,"thread_start_us":576,"threads_started":1}
I20260929 16:38:49.214186 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2728c4cf66a54d9f9043683cea4208d8" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7041ce305364d89971a8939bbcc2446" is_pre_election: true
I20260929 16:38:49.214331 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2728c4cf66a54d9f9043683cea4208d8" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda3126a74904629a6987ecb46cf86f0" is_pre_election: true
I20260929 16:38:49.214696 1683 raft_consensus.cc:2454] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 0.
I20260929 16:38:49.214764 1550 raft_consensus.cc:2454] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 0.
I20260929 16:38:49.215380 1350 leader_election.cc:302] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee, bda3126a74904629a6987ecb46cf86f0; no voters:
I20260929 16:38:49.215759 1752 raft_consensus.cc:2781] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:38:49.215883 1752 raft_consensus.cc:482] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:38:49.215981 1752 raft_consensus.cc:3037] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:49.218106 1752 raft_consensus.cc:504] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.218719 1752 leader_election.cc:288] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 election: Requested vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:38:49.219184 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2728c4cf66a54d9f9043683cea4208d8" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7041ce305364d89971a8939bbcc2446"
I20260929 16:38:49.219234 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2728c4cf66a54d9f9043683cea4208d8" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda3126a74904629a6987ecb46cf86f0"
I20260929 16:38:49.219408 1683 raft_consensus.cc:3037] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:49.219432 1550 raft_consensus.cc:3037] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:38:49.221494 1683 raft_consensus.cc:2454] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 1.
I20260929 16:38:49.221524 1550 raft_consensus.cc:2454] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 1.
I20260929 16:38:49.221873 1351 leader_election.cc:302] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee, a7041ce305364d89971a8939bbcc2446; no voters:
I20260929 16:38:49.222179 1752 raft_consensus.cc:2781] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:38:49.222601 1752 raft_consensus.cc:686] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ed7d0c19b4095be8e4662f72f20ee, State: Running, Role: LEADER
I20260929 16:38:49.223076 1752 consensus_queue.cc:237] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:38:49.227886 1270 catalog_manager.cc:5495] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1). New cstate: current_term: 1 leader_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } health_report { overall_health: UNKNOWN } } }
W20260929 16:38:49.253891 1462 tablet.cc:2369] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:38:49.257665 1729 tablet.cc:2369] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:38:49.259868 1596 tablet.cc:2369] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:38:49.272814 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.267390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.17.205.1:42513)
0929 16:38:49.267579 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.267586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.267662 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:38:49.267882 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.267887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.267965 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.268104 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:38:49.268469 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.269211 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.270703 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.271106 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.271237 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.271756 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:38:49.272268 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:38:49.272409 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.272483 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.272566 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:38:49.280776 1550 raft_consensus.cc:1261] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Refusing update from remote peer 1e2ed7d0c19b4095be8e4662f72f20ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:49.280778 1683 raft_consensus.cc:1261] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Refusing update from remote peer 1e2ed7d0c19b4095be8e4662f72f20ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:38:49.281664 1752 consensus_queue.cc:1035] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:49.281975 1757 consensus_queue.cc:1035] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:38:49.307190 1270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:49.307531 1270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:38:49.311194 1270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Sending DeleteTablet for 3 replicas of tablet 2728c4cf66a54d9f9043683cea4208d8
I20260929 16:38:49.312419 1396 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:49 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:34116
I20260929 16:38:49.312521 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1333
I20260929 16:38:49.313164 1663 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:49 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:46086
I20260929 16:38:49.313200 1530 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728c4cf66a54d9f9043683cea4208d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:38:49 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:33282
I20260929 16:38:49.331001 1772 tablet_replica.cc:330] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: stopping tablet replica
I20260929 16:38:49.332304 1772 raft_consensus.cc:2229] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:49.333863 1772 raft_consensus.cc:2258] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:49.334117 1773 tablet_replica.cc:330] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: stopping tablet replica
I20260929 16:38:49.334363 1770 tablet_replica.cc:330] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: stopping tablet replica
I20260929 16:38:49.334674 1773 raft_consensus.cc:2229] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:38:49.334779 1770 raft_consensus.cc:2229] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:38:49.335508 1770 raft_consensus.cc:2258] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:49.335592 1773 raft_consensus.cc:2258] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:38:49.338064 1772 ts_tablet_manager.cc:1905] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:49.338739 1773 ts_tablet_manager.cc:1905] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:49.339890 1770 ts_tablet_manager.cc:1905] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:38:49.345791 1773 ts_tablet_manager.cc:1918] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:49.346010 1773 log.cc:1196] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/2728c4cf66a54d9f9043683cea4208d8
I20260929 16:38:49.346587 1773 ts_tablet_manager.cc:1939] T 2728c4cf66a54d9f9043683cea4208d8 P bda3126a74904629a6987ecb46cf86f0: Deleting consensus metadata
Sep 29 16:38:49 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:38:49.348701 1257 catalog_manager.cc:4841] TS bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): tablet 2728c4cf66a54d9f9043683cea4208d8 (table test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]) successfully deleted
I20260929 16:38:49.351007 1772 ts_tablet_manager.cc:1918] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:49.351202 1772 log.cc:1196] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/2728c4cf66a54d9f9043683cea4208d8
I20260929 16:38:49.351749 1772 ts_tablet_manager.cc:1939] T 2728c4cf66a54d9f9043683cea4208d8 P a7041ce305364d89971a8939bbcc2446: Deleting consensus metadata
I20260929 16:38:49.353022 1770 ts_tablet_manager.cc:1918] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:38:49.353242 1770 log.cc:1196] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/2728c4cf66a54d9f9043683cea4208d8
I20260929 16:38:49.353628 1258 catalog_manager.cc:4841] TS a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): tablet 2728c4cf66a54d9f9043683cea4208d8 (table test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]) successfully deleted
I20260929 16:38:49.353789 1770 ts_tablet_manager.cc:1939] T 2728c4cf66a54d9f9043683cea4208d8 P 1e2ed7d0c19b4095be8e4662f72f20ee: Deleting consensus metadata
I20260929 16:38:49.355360 1258 catalog_manager.cc:4841] TS 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1:42513): tablet 2728c4cf66a54d9f9043683cea4208d8 (table test-table [id=8c4ef2c7c24945ecbfdb3ee1b3143c5a]) successfully deleted
I20260929 16:38:49.356968 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.313675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.17.205.1:42513)
0929 16:38:49.313874 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.313881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.336487 (+ 22606us) server_negotiation.cc:407] Connection header received
0929 16:38:49.336679 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.336684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.336770 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.336893 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:49.338940 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.340199 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.342245 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.342932 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.343045 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.348557 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:49.348576 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:49.348596 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:49.348661 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:49.352982 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.354261 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.354271 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.354274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.354483 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.355092 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.355097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.355100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.355580 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:49.355857 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.356503 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.356698 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:38:49.357656 1456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:49.470806 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1333
I20260929 16:38:49.486577 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1467
Sep 29 16:38:49 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:38:49.501706 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.487470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.17.205.2:43299)
0929 16:38:49.487660 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.487666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.487708 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:38:49.487934 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.487941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.488030 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.488126 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:49.489530 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.490327 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.491828 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.492269 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.492393 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.496503 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:49.496519 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:49.496538 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:49.496579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:49.499137 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.499937 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.499945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.499947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.500107 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.500411 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.500414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.500416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.500789 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:49.501010 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.501280 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.501452 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:38:49.502410 1590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:49.589666 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1467
I20260929 16:38:49.603824 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1600
Sep 29 16:38:49 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:38:49.618206 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:38:49.604569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.17.205.3:39499)
0929 16:38:49.604746 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:38:49.604751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:38:49.604768 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:38:49.604871 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:38:49.604875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:38:49.604938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:38:49.605030 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:38:49.606300 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.607014 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.608387 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:38:49.608784 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:38:49.608876 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:38:49.613227 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:38:49.613242 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:38:49.613259 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:38:49.613295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:38:49.615889 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.616508 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.616516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.616518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.616645 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:38:49.616931 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:38:49.616933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:38:49.616936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:38:49.617298 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:38:49.617521 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:38:49.617715 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:38:49.617885 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:38:49.618860 1723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:49.710440 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1600
I20260929 16:38:49.723824 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1240
I20260929 16:38:49.724887 1302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:38:49.833137 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1240
I20260929 16:39:09.850639 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--webserver_interface=127.17.205.62
--webserver_port=35041
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:10.185228 1778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:10.185534 1778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:10.185650 1778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:10.193094 1778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:39:10.193200 1778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:39:10.193238 1778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:39:10.193306 1778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:39:10.193377 1778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:10.193454 1778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:10.193502 1778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:39:10.193574 1778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:39:10.203929 1778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:44929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=35041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:10.205291 1778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:10.206671 1778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:10.229110 1784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:10.229142 1785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:10.229187 1787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:10.229904 1778 server_base.cc:1034] running on GCE node
I20260929 16:39:10.230822 1778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:10.232525 1778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:10.233755 1778 hybrid_clock.cc:648] HybridClock initialized: now 1790699950233668 us; error 57 us; skew 500 ppm
I20260929 16:39:10.234145 1778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:10 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:10.244027 1778 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:39:10.245816 1778 webserver.cc:460] Webserver started at http://127.17.205.62:35041/ using document root <none> and password file <none>
I20260929 16:39:10.246668 1778 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:10.246793 1778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:10.253022 1778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:39:10.256126 1794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:10.257547 1778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:10.257721 1778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "ecea2d18dba1416995e0ac979bc12425"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "e36e5741553d06ccb6bc7a6ab67393ee"
server_key_iv: "5674c5d063bfe757eabc5230da5a8519"
server_key_version: "encryptionkey@0"
I20260929 16:39:10.258762 1778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:10.274672 1778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:10.278733 1778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:10.279019 1778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:10.296815 1778 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:44929
I20260929 16:39:10.296829 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:44929 every 8 connection(s)
I20260929 16:39:10.298377 1778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:39:10.304441 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1778
I20260929 16:39:10.305192 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:42513
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=33133
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:39:10.305696 1846 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:39:10.309180 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap starting.
I20260929 16:39:10.328944 1846 log.cc:824] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:10.341301 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:39:10.341815 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap complete.
I20260929 16:39:10.350773 1846 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:10.351679 1846 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Initialized, Role: FOLLOWER
I20260929 16:39:10.352367 1846 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:10.352607 1846 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:39:10.352732 1846 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:39:10.352887 1846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:39:10.356845 1846 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:10.357360 1846 leader_election.cc:302] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecea2d18dba1416995e0ac979bc12425; no voters:
I20260929 16:39:10.357947 1846 leader_election.cc:288] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:39:10.358129 1850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:39:10.359048 1850 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 LEADER]: Becoming Leader. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Running, Role: LEADER
I20260929 16:39:10.359727 1850 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:10.359965 1846 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:39:10.362638 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecea2d18dba1416995e0ac979bc12425. Latest consensus state: current_term: 2 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:39:10.362982 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:10.363497 1851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:39:10.363767 1851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:10.372231 1858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:39:10.380964 1858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:39:10.382490 1858 catalog_manager.cc:1259] Loaded cluster ID: 2bb8417b976248fd96ac4e6fd36bb51f
I20260929 16:39:10.382682 1858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:39:10.387450 1858 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:39:10.389142 1858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Loaded TSK: 0
I20260929 16:39:10.427296 1858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Generated new TSK 1
I20260929 16:39:10.427866 1858 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:39:10.658135 1848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:10.658438 1848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:10.658571 1848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:10.666217 1848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:10.666352 1848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:10.666512 1848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:39:10.676573 1848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=33133
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:10.677872 1848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:10.679180 1848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:10.694052 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:10.693866 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:10.694743 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:10.695370 1848 server_base.cc:1034] running on GCE node
I20260929 16:39:10.696048 1848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:10.697021 1848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:10.698222 1848 hybrid_clock.cc:648] HybridClock initialized: now 1790699950698157 us; error 61 us; skew 500 ppm
I20260929 16:39:10.698549 1848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:10 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:10.701746 1848 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:39:10.703320 1848 webserver.cc:460] Webserver started at http://127.17.205.1:33133/ using document root <none> and password file <none>
I20260929 16:39:10.704020 1848 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:10.704151 1848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:10.709832 1848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:39:10.713130 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:10.714707 1848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:10.714896 1848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "1e2ed7d0c19b4095be8e4662f72f20ee"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "cc7238d42d70d1177f631a636d121d6a"
server_key_iv: "9d26a965c7fa9c13822d417fcc2a3737"
server_key_version: "encryptionkey@0"
I20260929 16:39:10.715991 1848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:10.753052 1848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:10.757437 1848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:10.757690 1848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:10.758671 1848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:10.760040 1848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:10.760172 1848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:10.760280 1848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:10.760345 1848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:10.791531 1848 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42513
I20260929 16:39:10.791584 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42513 every 8 connection(s)
I20260929 16:39:10.793145 1848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:39:10.798342 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 1848
I20260929 16:39:10.799129 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:43299
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=38369
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:39:10 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:10.819388 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:10.796386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:44907)
0929 16:39:10.797176 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:10.797220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:10.799514 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:10.801092 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:10.801103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:10.801805 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:10.802628 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:10.802655 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:10.804465 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:10.804470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:10.806268 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:10.806285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:10.807515 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:10.807524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:10.807824 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:10.808516 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:10.808546 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:10.810749 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:10.814988 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:10.814996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:10.815011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:10.815538 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:10.816056 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:10.816059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:10.816061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:10.816254 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:10.817157 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:10.817163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:10.817584 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 16:39:10.817821 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 16:39:10.818259 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":299,"threads_started":1}
I20260929 16:39:10.819855 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:10.796065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:44907 (local address 127.17.205.62:44929)
0929 16:39:10.796640 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:10.796653 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:10.797360 (+ 707us) server_negotiation.cc:407] Connection header received
0929 16:39:10.800117 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:10.800133 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:10.800534 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:10.800854 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:10.802929 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:10.804287 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:10.806435 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:10.807191 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:10.807418 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:10.811095 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:10.811109 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:10.811132 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:10.811185 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:10.814789 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:10.815681 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:10.815692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:10.815701 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:10.815948 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:10.816353 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:10.816358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:10.816361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:10.817022 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:10.817214 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:10.818107 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 16:39:10.818580 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260929 16:39:10.822765 1997 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:39:10.823069 1997 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:10.823820 1997 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:39:10.825966 1811 ts_manager.cc:194] Registered new tserver with Master: 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1:42513)
I20260929 16:39:10.827545 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:44907
W20260929 16:39:11.126433 2001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:11.126715 2001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:11.126840 2001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:11.134234 2001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:11.134347 2001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:11.134488 2001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:39:11.144416 2001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:43299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=38369
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:11.145699 2001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:11.146998 2001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:11.159891 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:11.159808 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:11.159840 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:11.160952 2001 server_base.cc:1034] running on GCE node
I20260929 16:39:11.161682 2001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:11.162808 2001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:11.164022 2001 hybrid_clock.cc:648] HybridClock initialized: now 1790699951163969 us; error 38 us; skew 500 ppm
I20260929 16:39:11.164368 2001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:11 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:11.167589 2001 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:39:11.169116 2001 webserver.cc:460] Webserver started at http://127.17.205.2:38369/ using document root <none> and password file <none>
I20260929 16:39:11.169824 2001 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:11.169977 2001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:11.175843 2001 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:39:11.178764 2016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.180181 2001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:11.180351 2001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "bda3126a74904629a6987ecb46cf86f0"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "de92540bb80867570b6edeefbb336fe2"
server_key_iv: "bf2920f346ba6cdc5797772e77e07709"
server_key_version: "encryptionkey@0"
I20260929 16:39:11.181397 2001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:11.204411 2001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:11.208999 2001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:11.209290 2001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:11.210402 2001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:11.211920 2001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:11.212028 2001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.212148 2001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:11.212224 2001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.244167 2001 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:43299
I20260929 16:39:11.244217 2128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:43299 every 8 connection(s)
I20260929 16:39:11.245646 2001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:39:11.255111 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2001
I20260929 16:39:11.255769 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:39499
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=42393
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:39:11 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:11.265758 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.248182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:51079 (local address 127.17.205.62:44929)
0929 16:39:11.248346 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.248357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.249466 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 16:39:11.251316 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.251321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.251417 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.251505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:11.253049 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.253765 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.255389 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.255941 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.256072 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.259539 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:11.259545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:11.259548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:11.259584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:11.262281 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.263316 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.263321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.263323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.263481 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.263899 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.263904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.263907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.264291 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:11.264450 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.265114 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.265380 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:39:11.266577 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.248625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:51079)
0929 16:39:11.249308 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.249346 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.251019 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.251759 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.251768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.252284 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:11.252871 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.252890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.253925 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.253928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.255165 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.255178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.256191 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.256199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.256437 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.257092 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:11.257121 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:11.259287 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:11.262491 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:11.262502 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:11.262522 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:11.263172 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:11.263595 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:11.263599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:11.263602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:11.263791 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:11.264402 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:11.264408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:11.264751 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.264982 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.265392 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":266,"threads_started":1}
I20260929 16:39:11.268154 2129 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:39:11.268543 2129 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:11.269493 2129 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:39:11.271162 1811 ts_manager.cc:194] Registered new tserver with Master: bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:39:11.272001 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:51079
W20260929 16:39:11.590145 2132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:11.590417 2132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:11.590536 2132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:11.597721 2132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:11.597842 2132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:11.597993 2132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:39:11.607738 2132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:39499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=42393
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:11.609069 2132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:11.610448 2132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:11.623462 2140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:11.623389 2138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:11.623490 2137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:11.624253 2132 server_base.cc:1034] running on GCE node
I20260929 16:39:11.624832 2132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:11.625769 2132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:11.626968 2132 hybrid_clock.cc:648] HybridClock initialized: now 1790699951626909 us; error 52 us; skew 500 ppm
I20260929 16:39:11.627315 2132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:11 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:11.630384 2132 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:39:11.631841 2132 webserver.cc:460] Webserver started at http://127.17.205.3:42393/ using document root <none> and password file <none>
I20260929 16:39:11.632544 2132 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:11.632669 2132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:11.638175 2132 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:39:11.640955 2147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.642282 2132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:39:11.642452 2132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "a7041ce305364d89971a8939bbcc2446"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "b4c35c16e094bf4bf735ec12de1737cd"
server_key_iv: "13c3da1fe6a939940aa97f683eeb09a7"
server_key_version: "encryptionkey@0"
I20260929 16:39:11.643488 2132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:11.668834 2132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:11.673777 2132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:11.674036 2132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:11.675259 2132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:11.677034 2132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:11.677122 2132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.677220 2132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:11.677415 2132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:11.707506 2132 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:39499
I20260929 16:39:11.707566 2259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:39499 every 8 connection(s)
I20260929 16:39:11.708989 2132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:39:11.710577 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2132
Sep 29 16:39:11 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:11.725996 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.711569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.17.205.62:44929)
0929 16:39:11.711768 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.711774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.712195 (+ 421us) server_negotiation.cc:407] Connection header received
0929 16:39:11.712382 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.712386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.712454 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.712548 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:11.714351 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.715441 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.716771 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.717280 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.717372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.720336 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:11.720342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:11.720347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:11.720394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:11.723339 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.723988 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.723992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.723995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.724127 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.724520 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.724523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.724525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.724788 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:11.725052 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.725385 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.725611 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":65}
I20260929 16:39:11.729923 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.711870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:51627 (local address 127.17.205.62:44929)
0929 16:39:11.712299 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.712303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.713143 (+ 840us) server_negotiation.cc:407] Connection header received
0929 16:39:11.715180 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.715184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.715280 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.715377 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:11.716865 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.717759 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.719405 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.720083 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.720208 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.724041 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:11.724046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:11.724050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:11.724092 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:11.726993 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.727836 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.727841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.727844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.727991 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.728430 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.728435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.728437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.728707 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:11.728815 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.729459 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.729622 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":290,"thread_start_us":306,"threads_started":1}
I20260929 16:39:11.730862 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.712206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:51627)
0929 16:39:11.713003 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.713047 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.714887 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.715634 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.715643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.716131 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:11.716672 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.716688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.717909 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.717915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.719273 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.719285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.720292 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.720300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.720592 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.721233 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:11.721263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:11.723812 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:11.727133 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:11.727144 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:11.727160 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:11.727726 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:11.728086 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:11.728089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:11.728092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:11.728307 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:11.728813 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:11.728819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:11.729158 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.729381 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.729785 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":329,"threads_started":1}
I20260929 16:39:11.732172 2260 heartbeater.cc:347] Connected to a master server at 127.17.205.62:44929
I20260929 16:39:11.732458 2260 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:11.733204 2260 heartbeater.cc:511] Master 127.17.205.62:44929 requested a full tablet report, sending...
I20260929 16:39:11.734589 1811 ts_manager.cc:194] Registered new tserver with Master: a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499)
I20260929 16:39:11.735410 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:51627
I20260929 16:39:11.740950 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:39:11.746882 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.741782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.17.205.62:44929)
0929 16:39:11.742011 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.742018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.742321 (+ 303us) server_negotiation.cc:407] Connection header received
0929 16:39:11.742493 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.742496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.742657 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.742781 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:39:11.743125 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.743858 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.745228 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.745625 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.745710 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.746149 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:39:11.746365 (+ 216us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:39:11.746640 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46820: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:39:11.747048 2263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46820: token expired
W20260929 16:39:11.747653 1731 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.205.62:44929)
I20260929 16:39:11.759116 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.748147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.17.205.62:44929)
0929 16:39:11.748293 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.748297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.748311 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:39:11.748378 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.748382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.748444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.748510 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:11.749787 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.750465 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.751728 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.752109 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.752195 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.754674 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:11.754679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:11.754683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:11.754712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:11.756999 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.757586 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.757590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.757593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.757723 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.758042 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.758045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.758047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.758341 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:11.758546 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.758730 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.758884 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:39:11.762670 1811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:39:11.769883 1811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:39:11.793732 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.781527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:43299 (local address 127.0.0.1:46838)
0929 16:39:11.783856 (+ 2329us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.783883 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.784055 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.785351 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.785358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.785395 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:11.785864 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.785879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.787997 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.788002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.790447 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.790462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.792724 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.792730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.792955 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.793015 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.793139 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.793332 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2189,"mutex_wait_us":1,"thread_start_us":1918,"threads_started":1}
I20260929 16:39:11.794655 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.782027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.17.205.2:43299)
0929 16:39:11.783230 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.783239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.784137 (+ 898us) server_negotiation.cc:407] Connection header received
0929 16:39:11.784243 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.784250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.784816 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.785019 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:11.786076 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.787824 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.790635 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.792243 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.792368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.792577 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.794144 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.794327 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":907,"threads_started":1}
I20260929 16:39:11.795760 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.781317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.17.205.3:39499)
0929 16:39:11.782708 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.782719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.783984 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 16:39:11.785367 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.785374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.785628 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.785794 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:11.786704 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.789445 (+ 2741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.791839 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.794357 (+ 2518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.794481 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.794699 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.795266 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.795456 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":995,"threads_started":1}
I20260929 16:39:11.796401 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.780986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:39499 (local address 127.0.0.1:37070)
0929 16:39:11.783847 (+ 2861us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.783883 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.784084 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.786104 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.786108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.786131 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:11.786511 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.786524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.789625 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.789630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.791669 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.791685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.794886 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.794892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.795050 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.795082 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.795948 (+ 866us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.796100 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2626,"thread_start_us":2503,"threads_started":1}
I20260929 16:39:11.799376 2064 tablet_service.cc:1467] Processing CreateTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 (DEFAULT_TABLE table=test-table [id=00658ac935664979990fda5eed4f7e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:11.801056 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:11.804352 2195 tablet_service.cc:1467] Processing CreateTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 (DEFAULT_TABLE table=test-table [id=00658ac935664979990fda5eed4f7e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:11.805126 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.784477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.17.205.1:42513)
0929 16:39:11.798961 (+ 14484us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.798969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.798998 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:39:11.799095 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.799102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.799383 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.799591 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:11.800084 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.801356 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.802814 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.804145 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.804273 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.804497 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.804680 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.804829 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14307,"thread_start_us":14149,"threads_started":1}
I20260929 16:39:11.805169 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.784448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33746)
0929 16:39:11.796637 (+ 12189us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.796657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.796763 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.799626 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.799634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.799668 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:11.799888 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.799899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.801508 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.801513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.802658 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.802670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.804314 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.804320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.804513 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.804547 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.804625 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.804767 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12092,"thread_start_us":15473,"threads_started":1}
I20260929 16:39:11.805928 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:11.808244 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 (DEFAULT_TABLE table=test-table [id=00658ac935664979990fda5eed4f7e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:11.809362 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:11.818338 2276 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Bootstrap starting.
I20260929 16:39:11.821560 2277 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Bootstrap starting.
I20260929 16:39:11.822574 2276 tablet_bootstrap.cc:654] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:11.822882 2278 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Bootstrap starting.
I20260929 16:39:11.824043 2276 log.cc:824] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:11.825160 2277 tablet_bootstrap.cc:654] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:11.826273 2278 tablet_bootstrap.cc:654] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:11.826326 2277 log.cc:824] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:11.827463 2276 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: No bootstrap required, opened a new log
I20260929 16:39:11.827486 2278 log.cc:824] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:11.827752 2276 ts_tablet_manager.cc:1397] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:39:11.829322 2277 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: No bootstrap required, opened a new log
I20260929 16:39:11.829583 2277 ts_tablet_manager.cc:1397] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:39:11.830451 2278 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: No bootstrap required, opened a new log
I20260929 16:39:11.830552 1997 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:39:11.830698 2278 ts_tablet_manager.cc:1397] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 16:39:11.838126 2276 raft_consensus.cc:348] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.838603 2276 raft_consensus.cc:374] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:11.838699 2276 raft_consensus.cc:729] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda3126a74904629a6987ecb46cf86f0, State: Initialized, Role: FOLLOWER
I20260929 16:39:11.839367 2276 consensus_queue.cc:260] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.840562 2129 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:39:11.840816 2278 raft_consensus.cc:348] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.841259 2276 ts_tablet_manager.cc:1428] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 16:39:11.841264 2278 raft_consensus.cc:374] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:11.841352 2278 raft_consensus.cc:729] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7041ce305364d89971a8939bbcc2446, State: Initialized, Role: FOLLOWER
I20260929 16:39:11.841904 2278 consensus_queue.cc:260] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.842317 2277 raft_consensus.cc:348] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.842792 2277 raft_consensus.cc:374] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:11.842897 2277 raft_consensus.cc:729] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ed7d0c19b4095be8e4662f72f20ee, State: Initialized, Role: FOLLOWER
I20260929 16:39:11.842969 2260 heartbeater.cc:503] Master 127.17.205.62:44929 was elected leader, sending a full tablet report...
I20260929 16:39:11.843600 2278 ts_tablet_manager.cc:1428] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260929 16:39:11.843639 2277 consensus_queue.cc:260] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.846004 2277 ts_tablet_manager.cc:1428] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260929 16:39:11.849643 2284 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:11.849836 2284 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.851291 2284 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:39:11.857828 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.851755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43731 (local address 127.17.205.3:39499)
0929 16:39:11.851959 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.851966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.852592 (+ 626us) server_negotiation.cc:407] Connection header received
0929 16:39:11.852758 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.852762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.852830 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.852913 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:11.853576 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.854284 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.856159 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.856912 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.857055 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.857126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.857492 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.857619 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:39:11.857825 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.851633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:43549 (local address 127.17.205.2:43299)
0929 16:39:11.851820 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.851825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.852443 (+ 618us) server_negotiation.cc:407] Connection header received
0929 16:39:11.852611 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.852616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.852693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.852784 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:11.853388 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.854167 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.856049 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.856899 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.857053 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.857134 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.857406 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.857540 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:39:11.858070 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.851742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:39499 (local address 127.17.205.1:43731)
0929 16:39:11.852506 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.852523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.852651 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.852956 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.852961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.853006 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:11.853429 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.853442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.854412 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.854416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.856017 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.856033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.857173 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.857179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.857323 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.857354 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.857451 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.857576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":22,"spinlock_wait_cycles":1280,"thread_start_us":423,"threads_started":1}
I20260929 16:39:11.857879 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.851631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:43299 (local address 127.17.205.1:43549)
0929 16:39:11.852323 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:11.852346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:11.852479 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:11.852805 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:11.852809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:11.852832 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:11.853253 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.853266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.854297 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.854303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.855889 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:11.855900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.857123 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.857129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:11.857266 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.857289 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:39:11.857357 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:39:11.857454 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":48,"spinlock_wait_cycles":29696,"thread_start_us":357,"threads_started":1}
I20260929 16:39:11.859093 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c3a639fb9c4f3f8f8c9c7e0f073fc9" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7041ce305364d89971a8939bbcc2446" is_pre_election: true
I20260929 16:39:11.859093 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c3a639fb9c4f3f8f8c9c7e0f073fc9" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda3126a74904629a6987ecb46cf86f0" is_pre_election: true
I20260929 16:39:11.859642 2084 raft_consensus.cc:2454] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 0.
I20260929 16:39:11.859655 2215 raft_consensus.cc:2454] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 0.
I20260929 16:39:11.860280 1887 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee, a7041ce305364d89971a8939bbcc2446; no voters:
I20260929 16:39:11.860628 2284 raft_consensus.cc:2781] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:39:11.860783 2284 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:39:11.860879 2284 raft_consensus.cc:3037] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:11.863150 2284 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.863759 2284 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 election: Requested vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
I20260929 16:39:11.864256 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c3a639fb9c4f3f8f8c9c7e0f073fc9" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7041ce305364d89971a8939bbcc2446"
I20260929 16:39:11.864279 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c3a639fb9c4f3f8f8c9c7e0f073fc9" candidate_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda3126a74904629a6987ecb46cf86f0"
I20260929 16:39:11.864500 2215 raft_consensus.cc:3037] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:11.864500 2084 raft_consensus.cc:3037] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:11.867858 2084 raft_consensus.cc:2454] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 1.
I20260929 16:39:11.867858 2215 raft_consensus.cc:2454] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ed7d0c19b4095be8e4662f72f20ee in term 1.
I20260929 16:39:11.868356 1886 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee, a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0; no voters:
I20260929 16:39:11.868711 2284 raft_consensus.cc:2781] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:11.869124 2284 raft_consensus.cc:686] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ed7d0c19b4095be8e4662f72f20ee, State: Running, Role: LEADER
I20260929 16:39:11.869630 2284 consensus_queue.cc:237] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:11.874030 1810 catalog_manager.cc:5495] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1). New cstate: current_term: 1 leader_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } health_report { overall_health: UNKNOWN } } }
I20260929 16:39:11.890558 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.885643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.17.205.1:42513)
0929 16:39:11.885821 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.885826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.885884 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:39:11.886071 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.886075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.886159 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.886305 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:39:11.886619 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.887385 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.888718 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.889141 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.889227 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.889656 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:39:11.890143 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:39:11.890247 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.890290 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.890365 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:39:11.898948 2215 raft_consensus.cc:1261] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Refusing update from remote peer 1e2ed7d0c19b4095be8e4662f72f20ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:11.898948 2084 raft_consensus.cc:1261] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Refusing update from remote peer 1e2ed7d0c19b4095be8e4662f72f20ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:11.900019 2287 consensus_queue.cc:1035] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:11.900348 2284 consensus_queue.cc:1035] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:11.913666 2290 mvcc.cc:205] Tried to move back new op lower bound from 7334707002966904832 to 7334707002861641728. Current Snapshot: MvccSnapshot[applied={T|T < 7334707002966904832}]
I20260929 16:39:11.925436 1810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:11.925697 1810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:11.929391 1810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Sending DeleteTablet for 3 replicas of tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:11.930636 2195 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:37070
I20260929 16:39:11.930642 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:33746
I20260929 16:39:11.930933 2064 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:11 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:46838
I20260929 16:39:11.931430 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1848
I20260929 16:39:11.944204 2299 tablet_replica.cc:330] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: stopping tablet replica
I20260929 16:39:11.945008 2299 raft_consensus.cc:2229] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:39:11.946028 2299 raft_consensus.cc:2258] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:11.946980 2301 tablet_replica.cc:330] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: stopping tablet replica
I20260929 16:39:11.947497 2301 raft_consensus.cc:2229] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:11.948072 2301 raft_consensus.cc:2258] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:11.948698 2300 tablet_replica.cc:330] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: stopping tablet replica
I20260929 16:39:11.949750 2299 ts_tablet_manager.cc:1905] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:11.950276 2300 raft_consensus.cc:2229] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:11.950460 2301 ts_tablet_manager.cc:1905] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:11.950917 2300 raft_consensus.cc:2258] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:11.953253 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:11.932633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.17.205.1:42513)
0929 16:39:11.932821 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:11.932839 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:11.932858 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:39:11.932913 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:11.932918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:11.933003 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:11.933096 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:11.934963 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.936216 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.938149 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:11.938866 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:11.938988 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:11.944808 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:11.944824 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:11.944852 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:11.944911 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:11.948918 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.949954 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.949964 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.949967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.950139 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:11.950593 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:11.950598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:11.950601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:11.951053 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:11.951379 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:11.952616 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0929 16:39:11.952879 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:39:11.954056 1992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:11.956570 2300 ts_tablet_manager.cc:1905] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:11.957790 2301 ts_tablet_manager.cc:1918] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:11.957935 2301 log.cc:1196] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:11.958451 2301 ts_tablet_manager.cc:1939] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P bda3126a74904629a6987ecb46cf86f0: Deleting consensus metadata
I20260929 16:39:11.960233 1796 catalog_manager.cc:4841] TS bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 (table test-table [id=00658ac935664979990fda5eed4f7e49]) successfully deleted
I20260929 16:39:11.962514 2300 ts_tablet_manager.cc:1918] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:11.962666 2300 log.cc:1196] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:11.963075 2300 ts_tablet_manager.cc:1939] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P a7041ce305364d89971a8939bbcc2446: Deleting consensus metadata
I20260929 16:39:11.964514 1797 catalog_manager.cc:4841] TS a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9 (table test-table [id=00658ac935664979990fda5eed4f7e49]) successfully deleted
I20260929 16:39:12.053318 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1848
I20260929 16:39:12.069124 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2001
W20260929 16:39:12.069123 1797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:42513 (error 108)
I20260929 16:39:12.070992 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.070583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33776)
0929 16:39:12.070730 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.070836 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:39:12.071204 1797 catalog_manager.cc:4584] TS 1e2ed7d0c19b4095be8e4662f72f20ee (127.17.205.1:42513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24c3a639fb9c4f3f8f8c9c7e0f073fc9: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
I20260929 16:39:12.087316 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.069938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.17.205.2:43299)
0929 16:39:12.070120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:12.070124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:12.070235 (+ 111us) server_negotiation.cc:407] Connection header received
0929 16:39:12.070446 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:12.070452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:12.070554 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:12.070662 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:12.072555 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:12.073283 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:12.074661 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:12.075087 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:12.075192 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:12.081552 (+ 6360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:12.081568 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:12.081585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:12.081621 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:12.084548 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:12.085387 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:12.085394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:12.085397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:12.085518 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:12.085900 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:12.085903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:12.085905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:12.086305 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:12.086563 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:12.086825 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:39:12.087017 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:39:12.087980 2124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:12.124864 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.124298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33790)
0929 16:39:12.124565 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.124683 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 16:39:12.125962 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.125524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33804)
0929 16:39:12.125687 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.125777 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:39:12.176806 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2001
I20260929 16:39:12.186626 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.186145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33814)
0929 16:39:12.186365 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.186477 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:39:12.187731 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.187349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33816)
0929 16:39:12.187496 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.187582 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 16:39:12.191200 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2132
I20260929 16:39:12.203980 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.191916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.17.205.3:39499)
0929 16:39:12.192071 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:12.192076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:12.192094 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:39:12.192227 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:12.192231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:12.192292 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:12.192369 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:12.193593 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:12.194328 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:12.195717 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:12.196126 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:12.196216 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:12.198599 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:12.198612 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:12.198630 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:12.198668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:12.201345 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:12.202079 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:12.202087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:12.202089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:12.202236 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:12.202605 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:12.202608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:12.202610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:12.202982 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:12.203194 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:12.203533 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 16:39:12.203705 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:39:12.204600 2255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:12.268567 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.268050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33824)
0929 16:39:12.268297 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.268422 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:39:12.269635 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:12.269228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:42513 (local address 127.0.0.1:33834)
0929 16:39:12.269417 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:12.269503 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:39:12.294483 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2132
I20260929 16:39:12.308568 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 1778
I20260929 16:39:12.309780 1841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:12.424867 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 1778
I20260929 16:39:32.443490 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--webserver_interface=127.17.205.62
--webserver_port=35041
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:32.771989 2307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:32.772296 2307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:32.772434 2307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:32.779795 2307 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:39:32.779918 2307 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:39:32.779979 2307 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:39:32.780046 2307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:39:32.780122 2307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:32.780215 2307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:32.780274 2307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:39:32.780349 2307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:39:32.790441 2307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:44929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=35041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:32.791817 2307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:32.793121 2307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:32.804638 2315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:32.804672 2313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:32.804703 2312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:32.805665 2307 server_base.cc:1034] running on GCE node
I20260929 16:39:32.806447 2307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:32.807894 2307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:32.809141 2307 hybrid_clock.cc:648] HybridClock initialized: now 1790699972809063 us; error 52 us; skew 500 ppm
I20260929 16:39:32.809489 2307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:32 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699972, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:32.816893 2307 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:39:32.819543 2307 webserver.cc:460] Webserver started at http://127.17.205.62:35041/ using document root <none> and password file <none>
I20260929 16:39:32.820500 2307 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:32.820626 2307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:32.828199 2307 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 16:39:32.831843 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:32.833730 2307 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:39:32.833931 2307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "ecea2d18dba1416995e0ac979bc12425"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "e36e5741553d06ccb6bc7a6ab67393ee"
server_key_iv: "5674c5d063bfe757eabc5230da5a8519"
server_key_version: "encryptionkey@0"
I20260929 16:39:32.835391 2307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:32.864415 2307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:32.868669 2307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:32.868939 2307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:32.885417 2307 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:44929
I20260929 16:39:32.885437 2373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:44929 every 8 connection(s)
I20260929 16:39:32.886863 2307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:39:32.893872 2374 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:39:32.896514 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2307
I20260929 16:39:32.897286 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap starting.
I20260929 16:39:32.897336 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:42513
--local_ip_for_outbound_sockets=127.17.205.1
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=33133
--webserver_interface=127.17.205.1
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:39:32.922084 2374 log.cc:824] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:32.943295 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:39:32.943980 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Bootstrap complete.
I20260929 16:39:32.957418 2374 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:32.958690 2374 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Initialized, Role: FOLLOWER
I20260929 16:39:32.959479 2374 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:32.959777 2374 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:39:32.959900 2374 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:39:32.960071 2374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:39:32.966087 2374 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:32.966792 2374 leader_election.cc:302] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecea2d18dba1416995e0ac979bc12425; no voters:
I20260929 16:39:32.967479 2374 leader_election.cc:288] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:39:32.967628 2378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:39:32.968670 2378 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [term 3 LEADER]: Becoming Leader. State: Replica: ecea2d18dba1416995e0ac979bc12425, State: Running, Role: LEADER
I20260929 16:39:32.969205 2378 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } }
I20260929 16:39:32.969664 2374 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:39:32.971515 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecea2d18dba1416995e0ac979bc12425. Latest consensus state: current_term: 3 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:39:32.971877 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:32.972226 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ecea2d18dba1416995e0ac979bc12425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea2d18dba1416995e0ac979bc12425" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 44929 } } }
I20260929 16:39:32.972522 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:32.984596 2384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:39:32.993409 2384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:39:32.994926 2384 catalog_manager.cc:1259] Loaded cluster ID: 2bb8417b976248fd96ac4e6fd36bb51f
I20260929 16:39:32.995047 2384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:39:32.999723 2384 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:39:33.001381 2384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Loaded TSK: 0
I20260929 16:39:33.001555 2384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ecea2d18dba1416995e0ac979bc12425: Loaded TSK: 1
F20260929 16:39:33.002727 2384 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2575a279ad google::LogMessage::Fail() at ??:0
@ 0x7f2575a2a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2575a274de google::LogMessage::Flush() at ??:0
@ 0x7f2575a28a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f257978b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f257977a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f258b4561c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f258b455783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
I20260929 16:39:33.027606 2396 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f258b4e12a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f258b4e10c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f258b52b5e9 std::function<>::operator()() at ??:0
@ 0x7f258b459cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f258b457c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f258b4df610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f258b4df439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2578673345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f257867691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2578676799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2578651eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f257b92b6db start_thread at ??:0
@ 0x7f25748c161f clone at ??:0
W20260929 16:39:33.325994 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:33.326282 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:33.326378 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:33.333665 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:33.333822 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:33.333990 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:39:33.343901 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=33133
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:33.345220 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:33.346470 2376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:33.359454 2422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:33.359477 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:33.359407 2424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:33.360528 2376 server_base.cc:1034] running on GCE node
I20260929 16:39:33.361133 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:33.362054 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:33.363291 2376 hybrid_clock.cc:648] HybridClock initialized: now 1790699973363218 us; error 45 us; skew 500 ppm
I20260929 16:39:33.363615 2376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:33 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:33.366621 2376 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:39:33.368142 2376 webserver.cc:460] Webserver started at http://127.17.205.1:33133/ using document root <none> and password file <none>
I20260929 16:39:33.368847 2376 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:33.368968 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:33.374480 2376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:39:33.377489 2431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:33.378893 2376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:33.379055 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "1e2ed7d0c19b4095be8e4662f72f20ee"
format_stamp: "Formatted at 2026-09-29 16:38:47 on dist-test-slave-3h01"
server_key: "cc7238d42d70d1177f631a636d121d6a"
server_key_iv: "9d26a965c7fa9c13822d417fcc2a3737"
server_key_version: "encryptionkey@0"
I20260929 16:39:33.380072 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:33.396348 2376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:33.400612 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:33.400877 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:33.401919 2376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:33.403824 2438 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 16:39:33.409529 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:39:33.409669 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:39:33.409797 2376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:39:33.413028 2376 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:39:33.413110 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:33.413496 2438 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Bootstrap starting.
I20260929 16:39:33.422379 2438 log.cc:824] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:33.436897 2438 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:39:33.437670 2438 tablet_bootstrap.cc:492] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Bootstrap complete.
I20260929 16:39:33.438251 2438 ts_tablet_manager.cc:1397] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20260929 16:39:33.446697 2438 raft_consensus.cc:348] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:33.448235 2438 raft_consensus.cc:729] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ed7d0c19b4095be8e4662f72f20ee, State: Initialized, Role: FOLLOWER
I20260929 16:39:33.449209 2438 consensus_queue.cc:260] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:33.450940 2438 ts_tablet_manager.cc:1428] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 16:39:33.454334 2376 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:42513
I20260929 16:39:33.454391 2546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:42513 every 8 connection(s)
I20260929 16:39:33.455644 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:39:33.456403 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2376
I20260929 16:39:33.457116 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:43299
--local_ip_for_outbound_sockets=127.17.205.2
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=38369
--webserver_interface=127.17.205.2
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:39:33.460703 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.458708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:38085)
0929 16:39:33.459547 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.459790 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":323,"threads_started":1}
W20260929 16:39:33.461440 2547 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:44929 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:44929: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
I20260929 16:39:33.462302 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.461968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:44773)
0929 16:39:33.462120 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.462194 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:39:33.463203 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.462875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:35051)
0929 16:39:33.463009 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.463082 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:39:33.463464 2547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:39:33.780429 2550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:33.780727 2550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:33.780821 2550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:33.788233 2550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:33.788378 2550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:33.788503 2550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:39:33.798331 2550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:43299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=38369
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:33.799746 2550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:33.801085 2550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:33.814215 2558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:33.814231 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:33.814173 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:33.814828 2550 server_base.cc:1034] running on GCE node
I20260929 16:39:33.815485 2550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:33.816421 2550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:33.817627 2550 hybrid_clock.cc:648] HybridClock initialized: now 1790699973817563 us; error 55 us; skew 500 ppm
I20260929 16:39:33.817957 2550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:33 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:33.821144 2550 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:39:33.822667 2550 webserver.cc:460] Webserver started at http://127.17.205.2:38369/ using document root <none> and password file <none>
I20260929 16:39:33.823434 2550 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:33.823549 2550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:33.829062 2550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:39:33.831920 2565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:33.833401 2550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:39:33.833559 2550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "bda3126a74904629a6987ecb46cf86f0"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "de92540bb80867570b6edeefbb336fe2"
server_key_iv: "bf2920f346ba6cdc5797772e77e07709"
server_key_version: "encryptionkey@0"
I20260929 16:39:33.834607 2550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:33.849179 2550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:33.853257 2550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:33.853513 2550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:33.854485 2550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:33.855850 2550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:33.855942 2550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:33.856050 2550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:33.856118 2550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:33.887091 2550 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:43299
I20260929 16:39:33.887146 2677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:43299 every 8 connection(s)
I20260929 16:39:33.888607 2550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:39:33.890475 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2550
I20260929 16:39:33.891114 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:39499
--local_ip_for_outbound_sockets=127.17.205.3
--tserver_master_addrs=127.17.205.62:44929
--webserver_port=42393
--webserver_interface=127.17.205.3
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:39:33.894168 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.891853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:36569)
0929 16:39:33.892691 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.892975 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":305,"threads_started":1}
W20260929 16:39:33.894937 2678 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:44929 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:44929: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
I20260929 16:39:33.895922 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.895520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:47285)
0929 16:39:33.895682 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.895784 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:39:33.896819 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:33.896460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:38829)
0929 16:39:33.896592 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:33.896679 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:39:33.897056 2678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:39:34.215242 2681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:34.215539 2681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:34.215641 2681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:34.222784 2681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:34.222901 2681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:34.223042 2681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:39:34.232846 2681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:39499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=42393
--tserver_master_addrs=127.17.205.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:34.234148 2681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:34.235553 2681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:34.248211 2686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:34.248188 2687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:34.248301 2689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:34.248842 2681 server_base.cc:1034] running on GCE node
I20260929 16:39:34.249434 2681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:34.250331 2681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:34.251536 2681 hybrid_clock.cc:648] HybridClock initialized: now 1790699974251481 us; error 52 us; skew 500 ppm
I20260929 16:39:34.251844 2681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:34 dist-test-slave-3h01 krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699974, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:34.254901 2681 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:39:34.256294 2681 webserver.cc:460] Webserver started at http://127.17.205.3:42393/ using document root <none> and password file <none>
I20260929 16:39:34.256942 2681 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:34.257040 2681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:34.262391 2681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:39:34.265237 2696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:34.266597 2681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:34.266729 2681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "a7041ce305364d89971a8939bbcc2446"
format_stamp: "Formatted at 2026-09-29 16:38:48 on dist-test-slave-3h01"
server_key: "b4c35c16e094bf4bf735ec12de1737cd"
server_key_iv: "13c3da1fe6a939940aa97f683eeb09a7"
server_key_version: "encryptionkey@0"
I20260929 16:39:34.267714 2681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:34.290907 2681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:34.294932 2681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:34.295151 2681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:34.296144 2681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:34.297456 2681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:34.297542 2681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:34.297618 2681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:34.297669 2681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:34.327363 2681 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:39499
I20260929 16:39:34.327473 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:39499 every 8 connection(s)
I20260929 16:39:34.328886 2681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:39:34.333983 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:34.332007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:54941)
0929 16:39:34.332762 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:34.333012 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":279,"threads_started":1}
W20260929 16:39:34.334736 2809 heartbeater.cc:644] Failed to heartbeat to 127.17.205.62:44929 (0 consecutive failures): Network error: Failed to ping master at 127.17.205.62:44929: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
I20260929 16:39:34.335245 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2681
I20260929 16:39:34.335652 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:34.335340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:43277)
0929 16:39:34.335475 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:34.335543 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 16:39:34.336449 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:34.336127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:51139)
0929 16:39:34.336263 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:34.336332 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:39:34.336719 2809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:39:34.464676 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:34.463979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:52109)
0929 16:39:34.464397 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:34.464488 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":189,"threads_started":1}
I20260929 16:39:34.898512 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:34.897577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:51957)
0929 16:39:34.898099 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:34.898233 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260929 16:39:35.083796 2815 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:35.084131 2815 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:35.085892 2815 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
Sep 29 16:39:35 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
Sep 29 16:39:35 dist-test-slave-3h01 krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:39:35.119015 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.086229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:39499 (local address 127.17.205.1:33875)
0929 16:39:35.099106 (+ 12877us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:35.099145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:35.099813 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:35.102532 (+ 2719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:35.102544 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:35.103521 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:35.104292 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:35.104313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.105846 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.105850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:35.107478 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:35.107491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.108486 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.108494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:35.108696 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:35.109390 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:35.109414 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:35.111760 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:35.115118 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:35.115128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:35.115143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:35.115763 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:35.116243 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:35.116247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:35.116249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:35.116484 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:35.117390 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:35.117398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:35.117755 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 16:39:35.118079 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 16:39:35.118602 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12698,"mutex_wait_us":97,"spinlock_wait_cycles":8064,"thread_start_us":12640,"threads_started":1}
I20260929 16:39:35.119140 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.086408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:33875 (local address 127.17.205.3:39499)
0929 16:39:35.098686 (+ 12278us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:35.098705 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:35.099408 (+ 703us) server_negotiation.cc:407] Connection header received
0929 16:39:35.101579 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:35.101587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:35.102007 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:35.102316 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:35.104498 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.105653 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.107612 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.108242 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.108416 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:35.112069 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:35.112076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:35.112092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:35.112130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:35.114970 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:35.115904 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:35.115913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:35.115922 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:35.116124 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:35.116595 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:35.116598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:35.116600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:35.117255 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:35.117415 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:35.118491 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0929 16:39:35.118825 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12118,"thread_start_us":12004,"threads_started":1}
I20260929 16:39:35.119349 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.086737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:58063 (local address 127.17.205.2:43299)
0929 16:39:35.098013 (+ 11276us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:35.098036 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:35.099327 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 16:39:35.101612 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:35.101621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:35.102008 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:35.102324 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:35.104494 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.105665 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.107612 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.108241 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.108417 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:35.112376 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:35.112383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:35.112396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:35.112435 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:35.115675 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:35.116428 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:35.116437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:35.116445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:35.116642 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:35.117018 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:35.117020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:35.117022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:35.117628 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:35.117843 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:35.118544 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 16:39:35.118959 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11024,"thread_start_us":11318,"threads_started":1}
I20260929 16:39:35.120126 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.086276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:43299 (local address 127.17.205.1:58063)
0929 16:39:35.099177 (+ 12901us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:35.099191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:35.099813 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:35.102532 (+ 2719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:35.102544 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:35.103521 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:35.104292 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:35.104313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.105838 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.105842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:35.107479 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:35.107491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:35.108436 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:35.108442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:35.108690 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:35.109447 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:35.109464 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:35.112130 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:35.115817 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:35.115822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:35.115825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:35.116291 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:35.116744 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:35.116749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:35.116752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:35.116925 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:35.117910 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:35.117914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:35.118059 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 16:39:35.118188 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:39:35.119773 (+ 1585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12812,"mutex_wait_us":1408,"spinlock_wait_cycles":86016,"thread_start_us":12481,"threads_started":1}
W20260929 16:39:35.122594 2434 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
W20260929 16:39:35.122881 2433 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:35.123024 2433 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee; no voters: a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0
I20260929 16:39:35.123474 2815 raft_consensus.cc:2726] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:39:35.338253 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.337307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:55901)
0929 16:39:35.337848 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:35.338007 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":218,"threads_started":1}
I20260929 16:39:35.465966 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.465523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:43305)
0929 16:39:35.465695 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:35.465822 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:39:35.900194 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:35.899439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:50789)
0929 16:39:35.899922 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:35.900006 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260929 16:39:36.339797 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:36.339112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:52675)
0929 16:39:36.339554 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:36.339639 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":244,"threads_started":1}
I20260929 16:39:36.467707 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:36.466843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:33371)
0929 16:39:36.467403 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:36.467511 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":257,"threads_started":1}
I20260929 16:39:36.901692 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:36.901019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:50709)
0929 16:39:36.901439 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:36.901520 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
I20260929 16:39:37.294597 2825 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:37.294768 2825 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:37.295430 2825 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
W20260929 16:39:37.296387 2434 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
W20260929 16:39:37.296567 2433 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:37.296684 2433 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee; no voters: a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0
I20260929 16:39:37.296927 2825 raft_consensus.cc:2726] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:39:37.341274 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:37.340575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:49599)
0929 16:39:37.341028 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:37.341111 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260929 16:39:37.469388 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:37.468602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:47293)
0929 16:39:37.469081 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:37.469181 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":221,"threads_started":1}
I20260929 16:39:37.903543 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:37.902700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:47155)
0929 16:39:37.903201 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:37.903347 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260929 16:39:38.343056 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:38.342183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:40187)
0929 16:39:38.342725 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:38.342839 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260929 16:39:38.471412 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:38.470466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:47367)
0929 16:39:38.471021 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:38.471138 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260929 16:39:38.905262 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:38.904520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:48043)
0929 16:39:38.904987 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:38.905077 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
I20260929 16:39:39.344874 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:39.344111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:40457)
0929 16:39:39.344571 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:39.344667 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
I20260929 16:39:39.473042 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:39.472302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:38381)
0929 16:39:39.472782 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:39.472878 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260929 16:39:39.516119 2834 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:39.516317 2834 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:39.517004 2834 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
W20260929 16:39:39.517980 2433 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
W20260929 16:39:39.518209 2434 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:39.518390 2434 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee; no voters: a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0
I20260929 16:39:39.518631 2834 raft_consensus.cc:2726] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:39:39.906826 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:39.906134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:35515)
0929 16:39:39.906558 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:39.906636 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260929 16:39:40.346624 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:40.345796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:41947)
0929 16:39:40.346358 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:40.346446 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":337,"threads_started":1}
I20260929 16:39:40.474905 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:40.474089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:60845)
0929 16:39:40.474580 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:40.474691 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260929 16:39:40.908607 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:40.907825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:39043)
0929 16:39:40.908333 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:40.908428 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":214,"threads_started":1}
I20260929 16:39:41.348764 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:41.347697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:56853)
0929 16:39:41.348384 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:41.348504 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":297,"threads_started":1}
I20260929 16:39:41.476522 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:41.475834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:40005)
0929 16:39:41.476264 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:41.476347 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260929 16:39:41.910331 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:41.909557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:46957)
0929 16:39:41.910041 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:41.910141 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260929 16:39:42.350481 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:42.349726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:35963)
0929 16:39:42.350207 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:42.350294 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260929 16:39:42.478309 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:42.477537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:45551)
0929 16:39:42.478043 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:42.478133 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":244,"threads_started":1}
I20260929 16:39:42.912076 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:42.911309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:54875)
0929 16:39:42.911814 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:42.911911 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":242,"threads_started":1}
I20260929 16:39:43.352141 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:43.351424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:57665)
0929 16:39:43.351888 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:43.351982 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":213,"threads_started":1}
I20260929 16:39:43.480144 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:43.479439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:44985)
0929 16:39:43.479891 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:43.479985 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260929 16:39:43.913803 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:43.913061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:60409)
0929 16:39:43.913530 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:43.913615 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260929 16:39:44.353915 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:44.353136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:44205)
0929 16:39:44.353646 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:44.353740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":253,"threads_started":1}
I20260929 16:39:44.433856 2849 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:44.434068 2849 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:44.434783 2849 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
W20260929 16:39:44.435684 2434 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
W20260929 16:39:44.435843 2433 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:44.436004 2433 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee; no voters: a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0
I20260929 16:39:44.436246 2849 raft_consensus.cc:2726] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:39:44.481968 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:44.481122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:55379)
0929 16:39:44.481667 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:44.481767 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":270,"threads_started":1}
I20260929 16:39:44.915503 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:44.914716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:38253)
0929 16:39:44.915193 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:44.915327 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260929 16:39:45.355598 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:45.354806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:60311)
0929 16:39:45.355332 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:45.355432 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260929 16:39:45.483682 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:45.482945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:53609)
0929 16:39:45.483426 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:45.483520 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":284,"threads_started":1}
I20260929 16:39:45.917392 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:45.916539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:58165)
0929 16:39:45.917089 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:45.917190 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
I20260929 16:39:46.357347 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:46.356589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:37485)
0929 16:39:46.357071 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:46.357162 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260929 16:39:46.485386 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:46.484579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:50679)
0929 16:39:46.485110 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:46.485208 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
I20260929 16:39:46.919029 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:46.918292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:52659)
0929 16:39:46.918760 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:46.918868 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260929 16:39:47.359078 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:47.358307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:56583)
0929 16:39:47.358818 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:47.358914 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260929 16:39:47.486891 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:47.486200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:58213)
0929 16:39:47.486623 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:47.486703 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260929 16:39:47.920658 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:47.919928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:47825)
0929 16:39:47.920376 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:47.920468 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260929 16:39:48.360713 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:48.359976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:39041)
0929 16:39:48.360441 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:48.360524 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":217,"threads_started":1}
I20260929 16:39:48.488721 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:48.488018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.1:39711)
0929 16:39:48.488459 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:48.488548 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260929 16:39:48.687553 2863 raft_consensus.cc:482] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:48.687804 2863 raft_consensus.cc:504] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7041ce305364d89971a8939bbcc2446" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 39499 } } peers { permanent_uuid: "1e2ed7d0c19b4095be8e4662f72f20ee" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 42513 } } peers { permanent_uuid: "bda3126a74904629a6987ecb46cf86f0" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 43299 } }
I20260929 16:39:48.688585 2863 leader_election.cc:288] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499), bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299)
W20260929 16:39:48.689465 2434 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7041ce305364d89971a8939bbcc2446 (127.17.205.3:39499): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
W20260929 16:39:48.689600 2433 leader_election.cc:341] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda3126a74904629a6987ecb46cf86f0 (127.17.205.2:43299): Not found: Tablet not found: 24c3a639fb9c4f3f8f8c9c7e0f073fc9
I20260929 16:39:48.689786 2433 leader_election.cc:302] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ed7d0c19b4095be8e4662f72f20ee; no voters: a7041ce305364d89971a8939bbcc2446, bda3126a74904629a6987ecb46cf86f0
I20260929 16:39:48.690062 2863 raft_consensus.cc:2726] T 24c3a639fb9c4f3f8f8c9c7e0f073fc9 P 1e2ed7d0c19b4095be8e4662f72f20ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:39:48.922727 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:48.921823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.2:52985)
0929 16:39:48.922347 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:48.922512 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
I20260929 16:39:49.345851 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2376
I20260929 16:39:49.362462 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:49.361729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:44929 (local address 127.17.205.3:38217)
0929 16:39:49.362188 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:49.362291 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.62:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
I20260929 16:39:49.363379 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:49.346863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.17.205.1:42513)
0929 16:39:49.347501 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:49.347511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:49.347545 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:39:49.347626 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:49.347635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:49.347962 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:49.348131 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:49.349495 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.350460 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.351827 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.352415 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.352533 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:49.357895 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:49.357908 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:49.357925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:49.357968 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:49.360536 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.361253 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.361261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.361263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.361404 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.361784 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.361788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.361790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.362160 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:49.362480 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:49.362887 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 16:39:49.363092 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
I20260929 16:39:49.364575 2542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:49.453773 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2376
I20260929 16:39:49.468748 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2550
I20260929 16:39:49.482045 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:49.469585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.17.205.2:43299)
0929 16:39:49.470059 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:49.470063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:49.470080 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:39:49.470137 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:49.470140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:49.470200 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:49.470287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:49.471828 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.472670 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.473924 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.474346 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.474442 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:49.477107 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:49.477112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:49.477116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:49.477151 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:49.479700 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.480303 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.480308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.480311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.480425 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.480758 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.480762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.480764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.481054 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:49.481259 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:49.481484 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:39:49.481655 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260929 16:39:49.482731 2673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:49.567333 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2550
I20260929 16:39:49.582314 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2681
I20260929 16:39:49.595610 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:49.583187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.17.205.3:39499)
0929 16:39:49.583747 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:49.583752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:49.583768 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:39:49.583838 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:49.583842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:49.583909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:49.583996 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:49.585216 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.586007 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.587181 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:49.587700 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:49.587797 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:49.590511 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:49.590515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:49.590518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:49.590551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:49.593203 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.593924 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.593928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.593930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.594051 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:49.594386 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:49.594389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:49.594390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:49.594642 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:49.594845 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:49.595104 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:39:49.595310 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":282,"threads_started":1}
I20260929 16:39:49.596362 2804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:49.680939 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2681
2026-09-29T16:39:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63935 ms)
[----------] 30 tests from SecurityITest (293731 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:39:49 dist-test-slave-3h01 krb5kdc[2874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:39:49 dist-test-slave-3h01 krb5kdc[2874](info): set up 2 sockets
Sep 29 16:39:49 dist-test-slave-3h01 krb5kdc[2874](info): commencing operation
krb5kdc: starting...
W20260929 16:39:50.691332 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:39:50 dist-test-slave-3h01 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:39:50.930805 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43157
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:51.263691 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:51.264006 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:51.264145 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:51.271275 2887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:39:51.271391 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:51.271469 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:51.271541 2887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:39:51.271601 2887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:39:51.281358 2887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:51.282778 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:51.284073 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:51.295080 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:51.295348 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:51.295114 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:51.296013 2887 server_base.cc:1034] running on GCE node
I20260929 16:39:51.296780 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:51.298172 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:51.299366 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790699991299323 us; error 29 us; skew 500 ppm
I20260929 16:39:51.299690 2887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:51 dist-test-slave-3h01 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699991, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:51.302743 2887 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:39:51.304262 2887 webserver.cc:460] Webserver started at http://127.17.205.62:45935/ using document root <none> and password file <none>
I20260929 16:39:51.304915 2887 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:51.305011 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:51.305263 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:51.308437 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "4b3bd4beda824323ab1ae33fb2ee06d6"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "ef8d8018da801a04252eff2722192211"
server_key_iv: "fa7216ff9667851fdb92fbc205f27420"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.309159 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "4b3bd4beda824323ab1ae33fb2ee06d6"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "ef8d8018da801a04252eff2722192211"
server_key_iv: "fa7216ff9667851fdb92fbc205f27420"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.314309 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:39:51.317744 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:51.319054 2887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:51.319255 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "4b3bd4beda824323ab1ae33fb2ee06d6"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "ef8d8018da801a04252eff2722192211"
server_key_iv: "fa7216ff9667851fdb92fbc205f27420"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.319437 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:51.335047 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:51.338951 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:51.339244 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:51.356626 2887 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:43157
I20260929 16:39:51.356647 2953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:43157 every 8 connection(s)
I20260929 16:39:51.358017 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:39:51.361765 2954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:51.364404 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2887
I20260929 16:39:51.364619 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:39:51.365410 18228 external_mini_cluster.cc:1428] Setting key c5a7aa32f0aa302e0f04d50d0833083b
I20260929 16:39:51.373575 2954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: Bootstrap starting.
I20260929 16:39:51.376612 2954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:51.377596 2954 log.cc:824] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:39:51 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:51.380565 2954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: No bootstrap required, opened a new log
I20260929 16:39:51.387415 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:51.367622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.17.205.62:43157)
0929 16:39:51.368411 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:51.368423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:51.368463 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:39:51.370145 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:51.370168 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:51.370584 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:51.370973 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:51.372507 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.373868 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.375173 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.375890 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.376054 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:51.380257 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:51.380266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:51.380280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:51.380321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:51.383350 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:51.384032 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:51.384039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:51.384045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:51.384209 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:51.384585 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:51.384588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:51.384590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:51.385153 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:51.385448 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:51.385917 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 16:39:51.386327 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":350,"threads_started":1}
I20260929 16:39:51.388311 2954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } }
I20260929 16:39:51.388661 2954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:51.388767 2954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3bd4beda824323ab1ae33fb2ee06d6, State: Initialized, Role: FOLLOWER
I20260929 16:39:51.389314 2954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } }
I20260929 16:39:51.389544 2954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:39:51.389664 2954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:39:51.389820 2954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:51.392071 2954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } }
I20260929 16:39:51.392535 2954 leader_election.cc:302] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b3bd4beda824323ab1ae33fb2ee06d6; no voters:
I20260929 16:39:51.393054 2954 leader_election.cc:288] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:39:51.393170 2959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:51.393797 2959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3bd4beda824323ab1ae33fb2ee06d6, State: Running, Role: LEADER
I20260929 16:39:51.394361 2959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } }
I20260929 16:39:51.394961 2954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:39:51.396917 2961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b3bd4beda824323ab1ae33fb2ee06d6. Latest consensus state: current_term: 1 leader_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } } }
I20260929 16:39:51.397007 2960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3bd4beda824323ab1ae33fb2ee06d6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43157 } } }
I20260929 16:39:51.397418 2961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:51.397435 2960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:51.398345 2967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:39:51.406783 2967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:39:51.417079 2967 catalog_manager.cc:1347] Generated new cluster ID: 39fa3b15f44e43e6943de63e6c045f1e
I20260929 16:39:51.417261 2967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:39:51.430557 2967 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:39:51.431936 2967 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:39:51.439325 2967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: Generated new TSK 0
I20260929 16:39:51.440136 2967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:39:51.507004 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:51.840410 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:51.840696 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:51.840828 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:51.848089 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:51.848217 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:51.848378 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:39:51.857961 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:51.859269 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:51.860612 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:51.873340 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:51.873378 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:51.873399 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:51.874115 2980 server_base.cc:1034] running on GCE node
I20260929 16:39:51.874675 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:51.875625 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:51.876823 2980 hybrid_clock.cc:648] HybridClock initialized: now 1790699991876782 us; error 31 us; skew 500 ppm
I20260929 16:39:51.877102 2980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:51 dist-test-slave-3h01 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699991, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:51.880105 2980 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:39:51.881515 2980 webserver.cc:460] Webserver started at http://127.17.205.1:42831/ using document root <none> and password file <none>
I20260929 16:39:51.882169 2980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:51.882289 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:51.882616 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:51.885522 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "333dc9d856124b98943ed89a592901f2"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "c23da908cf5e6cb2736bd779c6c44bc9"
server_key_iv: "da747678cfe841b906b0048a646d415a"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.886224 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "333dc9d856124b98943ed89a592901f2"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "c23da908cf5e6cb2736bd779c6c44bc9"
server_key_iv: "da747678cfe841b906b0048a646d415a"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.891315 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:39:51.894670 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:51.896150 2980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:39:51.896304 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "333dc9d856124b98943ed89a592901f2"
format_stamp: "Formatted at 2026-09-29 16:39:51 on dist-test-slave-3h01"
server_key: "c23da908cf5e6cb2736bd779c6c44bc9"
server_key_iv: "da747678cfe841b906b0048a646d415a"
server_key_version: "encryptionkey@0"
I20260929 16:39:51.896481 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:51.914503 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:51.918303 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:51.918531 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:51.919430 2980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:51.920696 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:51.920773 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:51.920862 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:51.920904 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:51.949815 2980 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:33981
I20260929 16:39:51.949847 3107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:33981 every 8 connection(s)
I20260929 16:39:51.951061 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:39:51.953671 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 2980
I20260929 16:39:51.953888 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:39:51.954528 18228 external_mini_cluster.cc:1428] Setting key e8178322e57446985941fd53ecee61e3
Sep 29 16:39:51 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699991, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:51.970727 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:51.953543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:50589 (local address 127.17.205.62:43157)
0929 16:39:51.953998 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:51.954004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:51.955074 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 16:39:51.956944 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:51.956950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:51.957034 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:51.957138 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:51.958642 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.959596 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.961109 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.961670 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.961787 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:51.965319 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:51.965324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:51.965327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:51.965361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:51.967794 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:51.968582 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:51.968585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:51.968588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:51.968700 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:51.969114 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:51.969117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:51.969119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:51.969377 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:51.969481 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:51.970289 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 16:39:51.970464 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":206,"threads_started":1}
I20260929 16:39:51.972115 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:51.954114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43157 (local address 127.17.205.1:50589)
0929 16:39:51.954898 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:51.954945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:51.956679 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:51.957358 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:51.957367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:51.957874 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:51.958465 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:51.958483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.959734 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.959739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:51.960973 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:51.960985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:51.961925 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:51.961930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:51.962136 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:51.962793 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:51.962820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:51.965091 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:51.967919 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:51.967928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:51.967944 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:51.968482 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:51.968788 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:51.968792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:51.968795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:51.969027 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:51.969489 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:51.969496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:51.969920 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 16:39:51.970241 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 16:39:51.970754 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
I20260929 16:39:51.973639 3108 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43157
I20260929 16:39:51.973963 3108 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:51.974885 3108 heartbeater.cc:511] Master 127.17.205.62:43157 requested a full tablet report, sending...
I20260929 16:39:51.977135 2919 ts_manager.cc:194] Registered new tserver with Master: 333dc9d856124b98943ed89a592901f2 (127.17.205.1:33981)
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:39:51.978613 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:39:52.005853 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:52.326992 3114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:52.327392 3114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:52.327538 3114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:52.335044 3114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:52.335183 3114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:52.335382 3114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:39:52.345806 3114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:52.347162 3114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:52.348479 3114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:52.362461 3120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:52.362624 3122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:52.362962 3119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:52.363736 3114 server_base.cc:1034] running on GCE node
I20260929 16:39:52.364368 3114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:52.365338 3114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:52.366528 3114 hybrid_clock.cc:648] HybridClock initialized: now 1790699992366470 us; error 48 us; skew 500 ppm
I20260929 16:39:52.366830 3114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:52 dist-test-slave-3h01 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699992, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:52.370121 3114 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:39:52.371719 3114 webserver.cc:460] Webserver started at http://127.17.205.2:46871/ using document root <none> and password file <none>
I20260929 16:39:52.372407 3114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:52.372538 3114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:52.372838 3114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:52.375805 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "71133b6a89654410ab03893f8b5b60ef"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "ac918dcccb16d4d06722e591e45e20df"
server_key_iv: "b783069baa4d6717a46a7dff7bd71ff1"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.376585 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "71133b6a89654410ab03893f8b5b60ef"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "ac918dcccb16d4d06722e591e45e20df"
server_key_iv: "b783069baa4d6717a46a7dff7bd71ff1"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.381780 3114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:39:52.385396 3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.386758 3114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:52.386916 3114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "71133b6a89654410ab03893f8b5b60ef"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "ac918dcccb16d4d06722e591e45e20df"
server_key_iv: "b783069baa4d6717a46a7dff7bd71ff1"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.387117 3114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:52.428092 3114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:52.432124 3114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:52.432430 3114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:52.433425 3114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:52.434785 3114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:52.434867 3114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.434986 3114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:52.435042 3114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.464915 3114 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:36741
I20260929 16:39:52.464958 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:36741 every 8 connection(s)
I20260929 16:39:52.466171 3114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:39:52.468230 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3114
I20260929 16:39:52.468391 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:39:52.468914 18228 external_mini_cluster.cc:1428] Setting key 86bba7e6e13cfefa4d08cfbbce740af5
Sep 29 16:39:52 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699992, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:52.487085 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:52.468788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:46987 (local address 127.17.205.62:43157)
0929 16:39:52.469003 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:52.469008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:52.470410 (+ 1402us) server_negotiation.cc:407] Connection header received
0929 16:39:52.472346 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:52.472352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:52.472433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:52.472555 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:52.474117 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.475070 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.476677 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.477233 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.477359 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:52.481175 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:52.481182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:52.481186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:52.481227 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:52.484179 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:52.484895 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:52.484900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:52.484903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:52.485045 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:52.485415 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:52.485419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:52.485422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:52.485759 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:52.485904 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:52.486543 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 16:39:52.486765 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:39:52.487996 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:52.469369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43157 (local address 127.17.205.2:46987)
0929 16:39:52.470240 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:52.470286 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:52.472053 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:52.472836 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:52.472845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:52.473360 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:52.473946 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:52.473966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.475274 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.475280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:52.476534 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:52.476547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.477431 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.477436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:52.477635 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:52.478328 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:52.478357 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:52.480925 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:52.484323 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:52.484331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:52.484345 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:52.484768 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:52.485132 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:52.485135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:52.485137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:52.485313 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:52.485876 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:52.485881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:52.486203 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 16:39:52.486423 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 16:39:52.486823 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":376,"threads_started":1}
I20260929 16:39:52.489495 3242 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43157
I20260929 16:39:52.489857 3242 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:52.490825 3242 heartbeater.cc:511] Master 127.17.205.62:43157 requested a full tablet report, sending...
I20260929 16:39:52.492651 2919 ts_manager.cc:194] Registered new tserver with Master: 71133b6a89654410ab03893f8b5b60ef (127.17.205.2:36741)
I20260929 16:39:52.493510 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:46987
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:39:52.522848 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:52.837982 3247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:52.838267 3247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:52.838393 3247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:52.845857 3247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:52.845971 3247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:52.846115 3247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:39:52.855806 3247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:52.857139 3247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:52.858435 3247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:52.871153 3252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:52.871241 3255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:52.871184 3253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:52.872061 3247 server_base.cc:1034] running on GCE node
I20260929 16:39:52.872699 3247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:52.873652 3247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:52.874851 3247 hybrid_clock.cc:648] HybridClock initialized: now 1790699992874778 us; error 65 us; skew 500 ppm
I20260929 16:39:52.875159 3247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:52 dist-test-slave-3h01 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699992, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:52.878212 3247 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:39:52.879724 3247 webserver.cc:460] Webserver started at http://127.17.205.3:37489/ using document root <none> and password file <none>
I20260929 16:39:52.880422 3247 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:52.880541 3247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:52.880848 3247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:52.883873 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "d594a5b9719b478794097d2d2832b577"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "82eb9b2d27a2f80f6d9243174e0c6de5"
server_key_iv: "caad6aa80b557db6fc2acd039835e3e3"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.884670 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "d594a5b9719b478794097d2d2832b577"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "82eb9b2d27a2f80f6d9243174e0c6de5"
server_key_iv: "caad6aa80b557db6fc2acd039835e3e3"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.890085 3247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 16:39:52.893646 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.895020 3247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:39:52.895174 3247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "d594a5b9719b478794097d2d2832b577"
format_stamp: "Formatted at 2026-09-29 16:39:52 on dist-test-slave-3h01"
server_key: "82eb9b2d27a2f80f6d9243174e0c6de5"
server_key_iv: "caad6aa80b557db6fc2acd039835e3e3"
server_key_version: "encryptionkey@0"
I20260929 16:39:52.895447 3247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:52.937088 3247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:52.942317 3247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:52.942639 3247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:52.943764 3247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:52.945492 3247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:52.945583 3247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.945708 3247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:52.945765 3247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:52.977061 3247 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:33031
I20260929 16:39:52.977095 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:33031 every 8 connection(s)
I20260929 16:39:52.978323 3247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:39:52.981732 3108 heartbeater.cc:503] Master 127.17.205.62:43157 was elected leader, sending a full tablet report...
I20260929 16:39:52.984047 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3247
I20260929 16:39:52.984225 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:39:52.984635 18228 external_mini_cluster.cc:1428] Setting key a8c1b1070d88d22547b8693d642647cf
Sep 29 16:39:52 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699992, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:52.997823 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:52.980848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:52589 (local address 127.17.205.62:43157)
0929 16:39:52.981024 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:52.981029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:52.982255 (+ 1226us) server_negotiation.cc:407] Connection header received
0929 16:39:52.984205 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:52.984210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:52.984327 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:52.984429 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:52.986127 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.986999 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.988632 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.989010 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.989118 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:52.992465 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:52.992472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:52.992475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:52.992509 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:52.994754 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:52.995516 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:52.995522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:52.995526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:52.995661 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:52.996077 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:52.996082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:52.996086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:52.996375 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:52.996505 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:52.997220 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 16:39:52.997456 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:39:52.998826 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:52.981285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43157 (local address 127.17.205.3:52589)
0929 16:39:52.982086 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:52.982132 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:52.983945 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:52.984667 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:52.984677 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:52.985301 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:52.985967 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:52.985988 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.987144 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.987147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:52.988494 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:52.988506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:52.989227 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:52.989234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:52.989466 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:52.990083 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:52.990110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:52.992256 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:52.994890 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:52.994899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:52.994917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:52.995378 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:52.995765 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:52.995770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:52.995774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:52.995966 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:52.996498 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:52.996506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:52.996856 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 16:39:52.997091 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 16:39:52.997629 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":295,"threads_started":1}
I20260929 16:39:53.000355 3375 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43157
I20260929 16:39:53.000633 3375 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:53.001330 3375 heartbeater.cc:511] Master 127.17.205.62:43157 requested a full tablet report, sending...
I20260929 16:39:53.002767 2919 ts_manager.cc:194] Registered new tserver with Master: d594a5b9719b478794097d2d2832b577 (127.17.205.3:33031)
I20260929 16:39:53.003616 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:52589
I20260929 16:39:53.012001 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:39:53.028828 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.016332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.17.205.62:43157)
0929 16:39:53.016487 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.016491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.016941 (+ 450us) server_negotiation.cc:407] Connection header received
0929 16:39:53.017109 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.017113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.017176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.017256 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:53.018692 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.019482 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.020834 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.021217 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.021326 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.023809 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:53.023814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:53.023818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:53.023852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:53.026192 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.026851 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.026857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.026859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.026993 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.027414 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.027419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.027422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.027740 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:53.027971 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.028312 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.028519 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:39:53.032577 2919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:39:53.039462 2919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:39:53.062523 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.051263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:33031 (local address 127.0.0.1:48510)
0929 16:39:53.053049 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:53.053067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:53.053171 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:53.055018 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:53.055028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:53.055072 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:53.055568 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.055583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.057445 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.057452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.059778 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.059795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.061564 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.061570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.061756 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.061802 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:39:53.061918 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:39:53.062106 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":4113,"threads_started":1}
I20260929 16:39:53.063534 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.051089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:36741 (local address 127.0.0.1:57264)
0929 16:39:53.052024 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:53.052069 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:53.052264 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:53.056788 (+ 4524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:53.056793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:53.056817 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:53.057330 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.057344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.059276 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.059281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.060966 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.060978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.062713 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.062720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.062885 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.062915 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:39:53.062980 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:39:53.063107 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":81,"thread_start_us":526,"threads_started":1}
I20260929 16:39:53.063813 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.051885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.17.205.3:33031)
0929 16:39:53.054299 (+ 2414us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.054309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.054348 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:39:53.054461 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.054470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.054757 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.054975 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:53.055830 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.057301 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.059964 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.061387 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.061495 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.061733 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.063288 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.063467 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2227,"thread_start_us":2051,"threads_started":1}
I20260929 16:39:53.064028 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.052520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33981 (local address 127.0.0.1:53548)
0929 16:39:53.055678 (+ 3158us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:53.055694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:53.055812 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:53.057041 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:53.057046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:53.057071 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:53.057515 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.057526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.059009 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.059014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.061141 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.061156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.063267 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.063274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.063429 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.063459 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:39:53.063551 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:39:53.063666 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2995,"mutex_wait_us":45,"thread_start_us":871,"threads_started":1}
I20260929 16:39:53.064064 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.051343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.17.205.2:36741)
0929 16:39:53.056303 (+ 4960us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.056312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.056339 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:39:53.056432 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.056439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.056613 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.056775 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:53.057543 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.059095 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.061107 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.062482 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.063351 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.063539 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.063609 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.063767 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4795,"thread_start_us":4659,"threads_started":1}
I20260929 16:39:53.064476 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.053170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.17.205.1:33981)
0929 16:39:53.053864 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.053874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.056264 (+ 2390us) server_negotiation.cc:407] Connection header received
0929 16:39:53.056372 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.056380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.056620 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.056795 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:53.057665 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.058854 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.061305 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.062726 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.062866 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.063098 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.064070 (+ 972us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.064244 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":353,"threads_started":1}
I20260929 16:39:53.067836 3310 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 (DEFAULT_TABLE table=test-table [id=71d7f4a331424b11a3714c9750777370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:53.068023 3043 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 (DEFAULT_TABLE table=test-table [id=71d7f4a331424b11a3714c9750777370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:53.068023 3177 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 (DEFAULT_TABLE table=test-table [id=71d7f4a331424b11a3714c9750777370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:53.069137 3310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb5dd20a2794378a6ad38542459d4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:53.069351 3177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb5dd20a2794378a6ad38542459d4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:53.069367 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb5dd20a2794378a6ad38542459d4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:53.080783 3393 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Bootstrap starting.
I20260929 16:39:53.083918 3394 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Bootstrap starting.
I20260929 16:39:53.085362 3393 tablet_bootstrap.cc:654] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:53.086763 3393 log.cc:824] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:53.087442 3394 tablet_bootstrap.cc:654] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:53.087913 3395 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Bootstrap starting.
I20260929 16:39:53.088917 3394 log.cc:824] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:53.091248 3395 tablet_bootstrap.cc:654] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:53.091318 3393 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: No bootstrap required, opened a new log
I20260929 16:39:53.091558 3393 ts_tablet_manager.cc:1397] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260929 16:39:53.091931 3394 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: No bootstrap required, opened a new log
I20260929 16:39:53.092190 3394 ts_tablet_manager.cc:1397] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 16:39:53.092444 3395 log.cc:824] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:53.096871 3395 tablet_bootstrap.cc:492] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: No bootstrap required, opened a new log
I20260929 16:39:53.097143 3395 ts_tablet_manager.cc:1397] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 16:39:53.099704 3393 raft_consensus.cc:348] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.100081 3393 raft_consensus.cc:374] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:53.100203 3393 raft_consensus.cc:729] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d594a5b9719b478794097d2d2832b577, State: Initialized, Role: FOLLOWER
I20260929 16:39:53.100980 3393 consensus_queue.cc:260] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.102161 3375 heartbeater.cc:503] Master 127.17.205.62:43157 was elected leader, sending a full tablet report...
I20260929 16:39:53.102339 3394 raft_consensus.cc:348] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.102805 3394 raft_consensus.cc:374] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:53.102821 3393 ts_tablet_manager.cc:1428] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260929 16:39:53.102895 3394 raft_consensus.cc:729] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333dc9d856124b98943ed89a592901f2, State: Initialized, Role: FOLLOWER
I20260929 16:39:53.103542 3394 consensus_queue.cc:260] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.107877 3395 raft_consensus.cc:348] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.108430 3395 raft_consensus.cc:374] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:53.108575 3395 raft_consensus.cc:729] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71133b6a89654410ab03893f8b5b60ef, State: Initialized, Role: FOLLOWER
I20260929 16:39:53.108731 3394 ts_tablet_manager.cc:1428] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 16:39:53.109289 3395 consensus_queue.cc:260] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.110535 3242 heartbeater.cc:503] Master 127.17.205.62:43157 was elected leader, sending a full tablet report...
I20260929 16:39:53.110821 3395 ts_tablet_manager.cc:1428] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260929 16:39:53.187664 3401 raft_consensus.cc:482] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:53.187996 3401 raft_consensus.cc:504] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.189879 3401 leader_election.cc:288] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d594a5b9719b478794097d2d2832b577 (127.17.205.3:33031), 333dc9d856124b98943ed89a592901f2 (127.17.205.1:33981)
I20260929 16:39:53.196777 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.190555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:33321 (local address 127.17.205.1:33981)
0929 16:39:53.190820 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.190825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.191443 (+ 618us) server_negotiation.cc:407] Connection header received
0929 16:39:53.191603 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.191609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.191693 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.191803 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:53.192427 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.193146 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.194999 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.195869 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.196029 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.196105 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.196431 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.196549 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:39:53.196890 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.190322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:33981 (local address 127.17.205.2:33321)
0929 16:39:53.191318 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:53.191332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:53.191462 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:53.191825 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:53.191828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:53.191861 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:53.192258 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.192266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.193277 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.193282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.194815 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.194846 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.196134 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.196142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.196267 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.196299 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:39:53.196383 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:39:53.196506 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":635,"threads_started":1}
I20260929 16:39:53.196928 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.190307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:58923 (local address 127.17.205.3:33031)
0929 16:39:53.190516 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.190522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.191163 (+ 641us) server_negotiation.cc:407] Connection header received
0929 16:39:53.191436 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.191443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.191521 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.191602 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:53.192403 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.193175 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.195073 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.195960 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.196147 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.196243 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.196535 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.196666 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:39:53.197096 3402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.190131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:33031 (local address 127.17.205.2:58923)
0929 16:39:53.191007 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:53.191033 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:53.191277 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:53.191651 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:53.191656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:53.191693 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:53.192245 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.192260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.193321 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.193331 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.194909 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:53.194922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.196190 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.196197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:53.196378 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.196403 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:39:53.196491 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:39:53.196618 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":5,"spinlock_wait_cycles":25344,"thread_start_us":508,"threads_started":1}
I20260929 16:39:53.198113 3330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5dd20a2794378a6ad38542459d4b6" candidate_uuid: "71133b6a89654410ab03893f8b5b60ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d594a5b9719b478794097d2d2832b577" is_pre_election: true
I20260929 16:39:53.198107 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5dd20a2794378a6ad38542459d4b6" candidate_uuid: "71133b6a89654410ab03893f8b5b60ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333dc9d856124b98943ed89a592901f2" is_pre_election: true
I20260929 16:39:53.198705 3063 raft_consensus.cc:2454] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71133b6a89654410ab03893f8b5b60ef in term 0.
I20260929 16:39:53.198715 3330 raft_consensus.cc:2454] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71133b6a89654410ab03893f8b5b60ef in term 0.
I20260929 16:39:53.199462 3131 leader_election.cc:302] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71133b6a89654410ab03893f8b5b60ef, d594a5b9719b478794097d2d2832b577; no voters:
I20260929 16:39:53.200103 3401 raft_consensus.cc:2781] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:39:53.200214 3401 raft_consensus.cc:482] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:39:53.200290 3401 raft_consensus.cc:3037] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:53.202498 3401 raft_consensus.cc:504] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.203109 3401 leader_election.cc:288] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [CANDIDATE]: Term 1 election: Requested vote from peers d594a5b9719b478794097d2d2832b577 (127.17.205.3:33031), 333dc9d856124b98943ed89a592901f2 (127.17.205.1:33981)
W20260929 16:39:53.203363 3109 tablet.cc:2369] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:39:53.203565 3330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5dd20a2794378a6ad38542459d4b6" candidate_uuid: "71133b6a89654410ab03893f8b5b60ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d594a5b9719b478794097d2d2832b577"
I20260929 16:39:53.203598 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5dd20a2794378a6ad38542459d4b6" candidate_uuid: "71133b6a89654410ab03893f8b5b60ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333dc9d856124b98943ed89a592901f2"
I20260929 16:39:53.203770 3063 raft_consensus.cc:3037] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:53.203758 3330 raft_consensus.cc:3037] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:53.205945 3063 raft_consensus.cc:2454] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71133b6a89654410ab03893f8b5b60ef in term 1.
I20260929 16:39:53.205950 3330 raft_consensus.cc:2454] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71133b6a89654410ab03893f8b5b60ef in term 1.
I20260929 16:39:53.206326 3131 leader_election.cc:302] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71133b6a89654410ab03893f8b5b60ef, d594a5b9719b478794097d2d2832b577; no voters:
I20260929 16:39:53.206609 3401 raft_consensus.cc:2781] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:53.207099 3401 raft_consensus.cc:686] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 1 LEADER]: Becoming Leader. State: Replica: 71133b6a89654410ab03893f8b5b60ef, State: Running, Role: LEADER
I20260929 16:39:53.207634 3401 consensus_queue.cc:237] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } }
I20260929 16:39:53.212167 2918 catalog_manager.cc:5495] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 71133b6a89654410ab03893f8b5b60ef (127.17.205.2). New cstate: current_term: 1 leader_uuid: "71133b6a89654410ab03893f8b5b60ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71133b6a89654410ab03893f8b5b60ef" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 } health_report { overall_health: UNKNOWN } } }
W20260929 16:39:53.217975 3243 tablet.cc:2369] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:39:53.229889 3376 tablet.cc:2369] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:39:53.256850 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.251505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.17.205.2:36741)
0929 16:39:53.251714 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.251721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.251818 (+ 97us) server_negotiation.cc:407] Connection header received
0929 16:39:53.252019 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.252022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.252102 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.252232 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:39:53.252605 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.253328 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.254741 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.255143 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.255269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.255760 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:39:53.256271 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:39:53.256410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.256490 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.256575 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:39:53.264818 3330 raft_consensus.cc:1261] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 1 FOLLOWER]: Refusing update from remote peer 71133b6a89654410ab03893f8b5b60ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:53.264824 3063 raft_consensus.cc:1261] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 1 FOLLOWER]: Refusing update from remote peer 71133b6a89654410ab03893f8b5b60ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:53.265728 3401 consensus_queue.cc:1035] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333dc9d856124b98943ed89a592901f2" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:53.266021 3404 consensus_queue.cc:1035] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d594a5b9719b478794097d2d2832b577" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:53.276985 3407 mvcc.cc:205] Tried to move back new op lower bound from 7334707172403159040 to 7334707172182306816. Current Snapshot: MvccSnapshot[applied={T|T < 7334707172403159040}]
I20260929 16:39:53.290310 2918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:53.290609 2918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:53.294296 2918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b3bd4beda824323ab1ae33fb2ee06d6: Sending DeleteTablet for 3 replicas of tablet 4eb5dd20a2794378a6ad38542459d4b6
I20260929 16:39:53.295341 3177 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:53 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:57264
I20260929 16:39:53.295466 3310 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:53 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:48510
I20260929 16:39:53.295734 3043 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eb5dd20a2794378a6ad38542459d4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:53 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260929 16:39:53.305258 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2980
I20260929 16:39:53.306775 3418 tablet_replica.cc:330] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: stopping tablet replica
I20260929 16:39:53.307415 3418 raft_consensus.cc:2229] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:53.308037 3418 raft_consensus.cc:2258] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:53.310793 3418 ts_tablet_manager.cc:1905] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:53.313046 3419 tablet_replica.cc:330] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: stopping tablet replica
I20260929 16:39:53.313674 3419 raft_consensus.cc:2229] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:53.314301 3419 raft_consensus.cc:2258] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:53.317288 3417 tablet_replica.cc:330] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: stopping tablet replica
I20260929 16:39:53.317674 3417 raft_consensus.cc:2229] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:39:53.318265 3417 raft_consensus.cc:2258] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:53.318686 3419 ts_tablet_manager.cc:1905] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:39:53 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:39:53.322335 3417 ts_tablet_manager.cc:1905] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:53.324035 3418 ts_tablet_manager.cc:1918] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:53.324231 3418 log.cc:1196] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/4eb5dd20a2794378a6ad38542459d4b6
I20260929 16:39:53.324846 3418 ts_tablet_manager.cc:1939] T 4eb5dd20a2794378a6ad38542459d4b6 P d594a5b9719b478794097d2d2832b577: Deleting consensus metadata
I20260929 16:39:53.326844 2904 catalog_manager.cc:4841] TS d594a5b9719b478794097d2d2832b577 (127.17.205.3:33031): tablet 4eb5dd20a2794378a6ad38542459d4b6 (table test-table [id=71d7f4a331424b11a3714c9750777370]) successfully deleted
I20260929 16:39:53.330157 3419 ts_tablet_manager.cc:1918] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:53.330417 3419 log.cc:1196] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/4eb5dd20a2794378a6ad38542459d4b6
I20260929 16:39:53.331018 3419 ts_tablet_manager.cc:1939] T 4eb5dd20a2794378a6ad38542459d4b6 P 333dc9d856124b98943ed89a592901f2: Deleting consensus metadata
I20260929 16:39:53.332422 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.306461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.17.205.1:33981)
0929 16:39:53.306668 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.306675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.308873 (+ 2198us) server_negotiation.cc:407] Connection header received
0929 16:39:53.311237 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.311243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.311374 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.311456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:53.315376 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.316533 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.318527 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.319153 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.319319 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.324524 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:53.324541 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:53.324559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:53.324604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:53.328517 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.329410 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.329420 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.329424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.329589 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.330063 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.330067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.330070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.330546 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:53.331203 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.331816 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.332034 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:39:53.333267 2905 catalog_manager.cc:4841] TS 333dc9d856124b98943ed89a592901f2 (127.17.205.1:33981): tablet 4eb5dd20a2794378a6ad38542459d4b6 (table test-table [id=71d7f4a331424b11a3714c9750777370]) successfully deleted
I20260929 16:39:53.333669 3417 ts_tablet_manager.cc:1918] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:53.333847 3417 log.cc:1196] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/4eb5dd20a2794378a6ad38542459d4b6
I20260929 16:39:53.334115 3103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:53.334441 3417 ts_tablet_manager.cc:1939] T 4eb5dd20a2794378a6ad38542459d4b6 P 71133b6a89654410ab03893f8b5b60ef: Deleting consensus metadata
I20260929 16:39:53.336809 2905 catalog_manager.cc:4841] TS 71133b6a89654410ab03893f8b5b60ef (127.17.205.2:36741): tablet 4eb5dd20a2794378a6ad38542459d4b6 (table test-table [id=71d7f4a331424b11a3714c9750777370]) successfully deleted
I20260929 16:39:53.422776 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2980
I20260929 16:39:53.437868 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3114
Sep 29 16:39:53 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:39:53.454452 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.438696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.17.205.2:36741)
0929 16:39:53.438867 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.438871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.438922 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:39:53.439110 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.439114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.439176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.439288 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:53.440862 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.441661 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.443591 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.444019 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.444108 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.448964 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:53.448976 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:53.448991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:53.449023 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:53.451841 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.452630 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.452637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.452639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.452777 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.453142 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.453145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.453147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.453483 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:53.453727 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.454008 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.454164 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:39:53.455168 3237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:53.568638 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3114
I20260929 16:39:53.584848 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3247
Sep 29 16:39:53 dist-test-slave-3h01 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:39:53.599289 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:53.585630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.17.205.3:33031)
0929 16:39:53.585859 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:53.585864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:53.585884 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:39:53.585976 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:53.585979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:53.586045 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:53.586126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:53.587393 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.588113 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.589406 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:53.589831 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:53.589933 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:53.594224 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:53.594240 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:53.594256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:53.594307 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:53.596945 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.597557 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.597564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.597566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.597708 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:53.597981 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:53.597984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:53.597986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:53.598342 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:53.598544 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:53.598798 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 16:39:53.598965 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:39:53.599964 3370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:53.688977 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3247
I20260929 16:39:53.702589 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 2887
I20260929 16:39:53.703819 2949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:53.813791 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 2887
2026-09-29T16:39:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4136 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:39:53Z Disabled control of system clock
I20260929 16:39:53.885418 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:46527
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:39:54.208909 3428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:54.209225 3428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:54.209335 3428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:54.216835 3428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:39:54.216949 3428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:54.216992 3428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:54.217069 3428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:39:54.217123 3428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:39:54.226583 3428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:46527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:54.227946 3428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:54.229748 3428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:54.241312 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:54.241315 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:54.241297 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:54.242287 3428 server_base.cc:1034] running on GCE node
I20260929 16:39:54.243101 3428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:54.244571 3428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:54.245797 3428 hybrid_clock.cc:648] HybridClock initialized: now 1790699994245739 us; error 49 us; skew 500 ppm
I20260929 16:39:54.246161 3428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:39:54.248394 3428 webserver.cc:460] Webserver started at http://127.17.205.62:36969/ using document root <none> and password file <none>
I20260929 16:39:54.249055 3428 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:54.249140 3428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:54.249399 3428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:54.252538 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "372073a525184fe7b73870334d509937"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "9ba97901426af69e796c4a3a86c4a47e"
server_key_iv: "b541709bf93faf4deccb0d05f1aac898"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.253315 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "372073a525184fe7b73870334d509937"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "9ba97901426af69e796c4a3a86c4a47e"
server_key_iv: "b541709bf93faf4deccb0d05f1aac898"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.258505 3428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:39:54.261899 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:54.263309 3428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:54.263458 3428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "372073a525184fe7b73870334d509937"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "9ba97901426af69e796c4a3a86c4a47e"
server_key_iv: "b541709bf93faf4deccb0d05f1aac898"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.263634 3428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:54.286274 3428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:54.287367 3428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:54.287645 3428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:54.304834 3428 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:46527
I20260929 16:39:54.304852 3493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:46527 every 8 connection(s)
I20260929 16:39:54.306181 3428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:39:54.307359 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3428
I20260929 16:39:54.307538 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:39:54.308081 18228 external_mini_cluster.cc:1428] Setting key b183532b6840dcb453466010acee8e54
I20260929 16:39:54.310886 3494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:54.321406 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:54.310026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.17.205.62:46527)
0929 16:39:54.311028 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:54.311040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:54.311084 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:39:54.312766 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:54.312789 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:54.313274 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:54.313731 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:54.314272 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.315643 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.317019 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.317666 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.317832 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:54.319004 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:54.319011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:54.319028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:54.319725 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:54.319810 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:54.319836 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:54.319947 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:54.320190 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 16:39:54.320490 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":339,"threads_started":1}
I20260929 16:39:54.322036 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: Bootstrap starting.
I20260929 16:39:54.325146 3494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:54.326236 3494 log.cc:824] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:54.328847 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: No bootstrap required, opened a new log
I20260929 16:39:54.336009 3494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } }
I20260929 16:39:54.336359 3494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:54.336450 3494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372073a525184fe7b73870334d509937, State: Initialized, Role: FOLLOWER
I20260929 16:39:54.337023 3494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } }
I20260929 16:39:54.337236 3494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:39:54.337360 3494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:39:54.337509 3494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:54.339818 3494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } }
I20260929 16:39:54.340289 3494 leader_election.cc:302] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372073a525184fe7b73870334d509937; no voters:
I20260929 16:39:54.340878 3494 leader_election.cc:288] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:39:54.341058 3499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:54.341756 3499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [term 1 LEADER]: Becoming Leader. State: Replica: 372073a525184fe7b73870334d509937, State: Running, Role: LEADER
I20260929 16:39:54.342307 3499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } }
I20260929 16:39:54.343088 3494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:39:54.345352 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372073a525184fe7b73870334d509937. Latest consensus state: current_term: 1 leader_uuid: "372073a525184fe7b73870334d509937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } } }
I20260929 16:39:54.345500 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372073a525184fe7b73870334d509937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372073a525184fe7b73870334d509937" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 46527 } } }
I20260929 16:39:54.345788 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:54.345791 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:54.348019 3506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:39:54.353117 3506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:39:54.361959 3506 catalog_manager.cc:1347] Generated new cluster ID: 4e44eb0ba7bb42cfa010964771cb3dd6
I20260929 16:39:54.362083 3506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:39:54.388811 3506 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:39:54.390215 3506 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:39:54.399850 3506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: Generated new TSK 0
I20260929 16:39:54.400769 3506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:39:54.405509 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--builtin_ntp_servers=127.17.205.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:39:54.735740 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:54.736020 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:54.736131 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:54.743613 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:54.743721 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:54.743857 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:39:54.753175 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:54.754485 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:54.756356 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:54.769119 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:54.769158 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:54.769179 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:54.769840 3518 server_base.cc:1034] running on GCE node
I20260929 16:39:54.770503 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:54.771571 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:54.772799 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790699994772738 us; error 48 us; skew 500 ppm
I20260929 16:39:54.773130 3518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:39:54.780629 3518 webserver.cc:460] Webserver started at http://127.17.205.1:38127/ using document root <none> and password file <none>
I20260929 16:39:54.781366 3518 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:54.781466 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:54.781783 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:54.784881 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "1f783d2ddf6a4677800ce363002de8d5"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "47e240e5961fb59bc826b4abb584bfbe"
server_key_iv: "bfe56f254b199834f132a4249a9fab59"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.785663 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "1f783d2ddf6a4677800ce363002de8d5"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "47e240e5961fb59bc826b4abb584bfbe"
server_key_iv: "bfe56f254b199834f132a4249a9fab59"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.791038 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:39:54.794611 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:54.796048 3518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:54.796203 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "1f783d2ddf6a4677800ce363002de8d5"
format_stamp: "Formatted at 2026-09-29 16:39:54 on dist-test-slave-3h01"
server_key: "47e240e5961fb59bc826b4abb584bfbe"
server_key_iv: "bfe56f254b199834f132a4249a9fab59"
server_key_version: "encryptionkey@0"
I20260929 16:39:54.796414 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:54.812974 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:54.813825 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:54.814110 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:54.815093 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:54.816591 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:54.816691 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:54.816802 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:54.816870 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:54.847065 3518 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:44177
I20260929 16:39:54.847111 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:44177 every 8 connection(s)
I20260929 16:39:54.848570 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:39:54.858441 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3518
I20260929 16:39:54.858613 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:39:54.859133 18228 external_mini_cluster.cc:1428] Setting key 6dc86acfbc359fb1e20c9e819fae9594
I20260929 16:39:54.861333 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:54.851071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:39421 (local address 127.17.205.62:46527)
0929 16:39:54.851504 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:54.851509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:54.852736 (+ 1227us) server_negotiation.cc:407] Connection header received
0929 16:39:54.854636 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:54.854642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:54.854722 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:54.854825 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:54.855983 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.856669 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.858268 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.858771 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.858900 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:54.860519 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:54.860526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:54.860530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:54.860551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:54.860603 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:54.860618 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:54.860702 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:54.860907 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:39:54.861005 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260929 16:39:54.862331 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--builtin_ntp_servers=127.17.205.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:39:54.862650 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:54.851708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46527 (local address 127.17.205.1:39421)
0929 16:39:54.852581 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:54.852623 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:54.854363 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:54.855083 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:54.855093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:54.855187 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:54.855816 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:54.855835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.856826 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.856830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:54.858108 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:54.858124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:54.859025 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:54.859033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:54.859374 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:54.860172 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:39:54.860175 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:54.860229 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:39:54.860336 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:54.860708 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:54.860714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:54.860720 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:39:54.860854 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:39:54.861241 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260929 16:39:54.864154 3645 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46527
I20260929 16:39:54.864500 3645 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:54.865397 3645 heartbeater.cc:511] Master 127.17.205.62:46527 requested a full tablet report, sending...
I20260929 16:39:54.867590 3459 ts_manager.cc:194] Registered new tserver with Master: 1f783d2ddf6a4677800ce363002de8d5 (127.17.205.1:44177)
I20260929 16:39:54.869153 3459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.1:39421
W20260929 16:39:55.182783 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:55.183063 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:55.183151 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:55.190527 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:55.190644 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:55.190761 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:39:55.200054 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:55.201339 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:55.203166 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:55.216126 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:55.216167 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:55.216147 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:55.216845 3649 server_base.cc:1034] running on GCE node
I20260929 16:39:55.217511 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:55.218511 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:55.219748 3649 hybrid_clock.cc:648] HybridClock initialized: now 1790699995219663 us; error 49 us; skew 500 ppm
I20260929 16:39:55.220063 3649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:39:55.222483 3649 webserver.cc:460] Webserver started at http://127.17.205.2:38315/ using document root <none> and password file <none>
I20260929 16:39:55.223186 3649 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:55.223315 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:55.223614 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:55.226485 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "b88fcba94b36443d897685286fddde31"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "bf38967d4871bb91efb2ff1682a5cb6c"
server_key_iv: "e057784a19afe7317afc3ce66f6d6bb9"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.227300 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "b88fcba94b36443d897685286fddde31"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "bf38967d4871bb91efb2ff1682a5cb6c"
server_key_iv: "e057784a19afe7317afc3ce66f6d6bb9"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.232372 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:39:55.235733 3663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:55.237044 3649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:39:55.237190 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "b88fcba94b36443d897685286fddde31"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "bf38967d4871bb91efb2ff1682a5cb6c"
server_key_iv: "e057784a19afe7317afc3ce66f6d6bb9"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.237401 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:55.259526 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:55.260372 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:55.260656 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:55.261613 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:55.263155 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:55.263271 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:39:55.263384 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:55.263454 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:55.292435 3649 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38759
I20260929 16:39:55.292491 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38759 every 8 connection(s)
I20260929 16:39:55.293689 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:39:55.296542 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3649
I20260929 16:39:55.296720 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:39:55.297274 18228 external_mini_cluster.cc:1428] Setting key 9512bc57625b91bbc598d53ca88fe146
I20260929 16:39:55.300276 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--builtin_ntp_servers=127.17.205.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:39:55.306805 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.296239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:45287 (local address 127.17.205.62:46527)
0929 16:39:55.296409 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.296414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.298204 (+ 1790us) server_negotiation.cc:407] Connection header received
0929 16:39:55.300414 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.300419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.300498 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.300611 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:55.301709 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.302460 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.304078 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.304546 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.304660 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.305989 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:55.305994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:55.305997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:55.306016 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:55.306048 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:55.306062 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:55.306134 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.306351 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.306460 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:39:55.308079 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.296992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46527 (local address 127.17.205.2:45287)
0929 16:39:55.298032 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.298094 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.300138 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.300856 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.300864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.300960 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:55.301545 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.301561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.302605 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.302610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.303924 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.303936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.304766 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.304773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.305002 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.305659 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:39:55.305662 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:55.305704 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:39:55.305792 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:55.306150 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:55.306161 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:55.306170 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.306334 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.306754 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":319,"threads_started":1}
I20260929 16:39:55.309640 3776 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46527
I20260929 16:39:55.309970 3776 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:55.310868 3776 heartbeater.cc:511] Master 127.17.205.62:46527 requested a full tablet report, sending...
I20260929 16:39:55.312515 3459 ts_manager.cc:194] Registered new tserver with Master: b88fcba94b36443d897685286fddde31 (127.17.205.2:38759)
I20260929 16:39:55.313601 3459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.2:45287
W20260929 16:39:55.623487 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:55.623796 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:55.623927 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:55.631182 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:55.631345 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:55.631500 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:39:55.640940 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:55.642279 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:55.644198 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:55.657106 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:55.657135 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:55.657157 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:55.657833 3779 server_base.cc:1034] running on GCE node
I20260929 16:39:55.658470 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:55.659472 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:55.660723 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790699995660641 us; error 49 us; skew 500 ppm
I20260929 16:39:55.661062 3779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:39:55.663390 3779 webserver.cc:460] Webserver started at http://127.17.205.3:39353/ using document root <none> and password file <none>
I20260929 16:39:55.664088 3779 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:55.664187 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:55.664492 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:55.667539 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "15a73416cb6a436c85fb52c0990352f5"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "23dbd2f320ba7fb1bd7991afc30ae000"
server_key_iv: "e850c3c1cc6e30510cd99e740f384237"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.668351 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "15a73416cb6a436c85fb52c0990352f5"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "23dbd2f320ba7fb1bd7991afc30ae000"
server_key_iv: "e850c3c1cc6e30510cd99e740f384237"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.674880 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:39:55.678439 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:55.679814 3779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:55.679975 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "15a73416cb6a436c85fb52c0990352f5"
format_stamp: "Formatted at 2026-09-29 16:39:55 on dist-test-slave-3h01"
server_key: "23dbd2f320ba7fb1bd7991afc30ae000"
server_key_iv: "e850c3c1cc6e30510cd99e740f384237"
server_key_version: "encryptionkey@0"
I20260929 16:39:55.680171 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:55.695319 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:55.696168 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:55.696450 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:55.697400 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:55.698832 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:55.698927 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:55.699036 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:55.699103 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:55.728564 3779 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:43055
I20260929 16:39:55.728610 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:43055 every 8 connection(s)
I20260929 16:39:55.729842 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:39:55.736627 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3779
I20260929 16:39:55.740872 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:39:55.741374 18228 external_mini_cluster.cc:1428] Setting key 09f1f8d90a90559b9753bb85e920ca2a
I20260929 16:39:55.743538 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.732389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:38879 (local address 127.17.205.62:46527)
0929 16:39:55.732566 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.732571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.733969 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 16:39:55.735882 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.735887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.735967 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.736053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:55.737521 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.738271 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.740578 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.741137 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.741241 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.742713 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:55.742718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:55.742721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:55.742738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:55.742771 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:55.742782 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:55.742864 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.743084 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.743189 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:39:55.744659 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.732922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:46527 (local address 127.17.205.3:38879)
0929 16:39:55.733813 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.733850 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.735607 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.736335 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.736347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.736471 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:55.737323 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.737350 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.738424 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.738429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.740422 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.740439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.741367 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.741375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.741666 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.742385 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:39:55.742388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:55.742441 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:39:55.742522 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:55.742879 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:55.742890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:55.742898 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.743054 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.743526 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260929 16:39:55.745867 3906 heartbeater.cc:347] Connected to a master server at 127.17.205.62:46527
I20260929 16:39:55.746121 3906 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:55.746829 3906 heartbeater.cc:511] Master 127.17.205.62:46527 requested a full tablet report, sending...
I20260929 16:39:55.748258 3459 ts_manager.cc:194] Registered new tserver with Master: 15a73416cb6a436c85fb52c0990352f5 (127.17.205.3:43055)
I20260929 16:39:55.749056 3459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.3:38879
I20260929 16:39:55.756582 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:39:55.765806 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.760332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.17.205.62:46527)
0929 16:39:55.760526 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.760530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.760901 (+ 371us) server_negotiation.cc:407] Connection header received
0929 16:39:55.761036 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.761040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.761116 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.761205 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:55.761615 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.762300 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.763625 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.764074 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.764205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.765166 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:55.765174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:55.765191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:55.765216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:55.765262 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:55.765281 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:55.765378 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.765466 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.765556 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:39:55.769572 3459 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:39:55.776312 3459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:39:55.808964 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.788713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:43055 (local address 127.0.0.1:56574)
0929 16:39:55.799740 (+ 11027us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.799805 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.800040 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.801255 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.801264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.801297 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:55.801719 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.801732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.803830 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.803835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.806137 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.806151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.808026 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.808032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.808245 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.808365 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.808461 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.808600 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10170,"mutex_wait_us":30,"thread_start_us":15223,"threads_started":1}
I20260929 16:39:55.809660 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.788969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.17.205.3:43055)
0929 16:39:55.798684 (+ 9715us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.798695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.800259 (+ 1564us) server_negotiation.cc:407] Connection header received
0929 16:39:55.800392 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.800402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.800704 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.800911 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:55.801954 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.803662 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.806341 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.807823 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.808713 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.809048 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.809129 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.809303 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9493,"thread_start_us":9305,"threads_started":1}
I20260929 16:39:55.813968 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.789388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.17.205.1:44177)
0929 16:39:55.805854 (+ 16466us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.805863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.805891 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:39:55.805981 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.805988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.806230 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.807108 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:55.807361 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.808693 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.811479 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.812887 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.813021 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.813258 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.813467 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.813634 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16298,"thread_start_us":16304,"threads_started":1}
I20260929 16:39:55.813967 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.788909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43402)
0929 16:39:55.803734 (+ 14825us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.803753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.803853 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.806451 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.806459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.806507 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:55.806932 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.806945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.808846 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.808851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.811249 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.811264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.813094 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.813100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.813273 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.813307 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.813406 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.813566 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14680,"mutex_wait_us":34,"spinlock_wait_cycles":4992,"thread_start_us":9829,"threads_started":1}
I20260929 16:39:55.814527 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.800189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.17.205.2:38759)
0929 16:39:55.805143 (+ 4954us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.805153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.805734 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:39:55.805841 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.805850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.806107 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.806277 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:55.806881 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.807975 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.811107 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.812912 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.813024 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.813256 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.814059 (+ 803us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.814218 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4777,"thread_start_us":4655,"threads_started":1}
I20260929 16:39:55.813951 3841 tablet_service.cc:1467] Processing CreateTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 (DEFAULT_TABLE table=test-table [id=2482f8db343f4807a07eb8e8f6bfdedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:55.814917 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.804291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38759 (local address 127.0.0.1:60988)
0929 16:39:55.805486 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.805503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.805620 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.806294 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.806299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.806323 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:55.806692 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.806702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.808134 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.808139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.810895 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.810911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.813377 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.813384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.813599 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.813630 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.813712 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.813858 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":109,"thread_start_us":894,"threads_started":1}
I20260929 16:39:55.815800 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:55.817831 3711 tablet_service.cc:1467] Processing CreateTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 (DEFAULT_TABLE table=test-table [id=2482f8db343f4807a07eb8e8f6bfdedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:55.817747 3580 tablet_service.cc:1467] Processing CreateTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 (DEFAULT_TABLE table=test-table [id=2482f8db343f4807a07eb8e8f6bfdedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:55.818852 3711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:55.819057 3580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:55.829871 3924 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Bootstrap starting.
I20260929 16:39:55.833560 3926 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Bootstrap starting.
I20260929 16:39:55.834270 3924 tablet_bootstrap.cc:654] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:55.835126 3925 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Bootstrap starting.
I20260929 16:39:55.835770 3924 log.cc:824] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:55.837034 3926 tablet_bootstrap.cc:654] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:55.838146 3925 tablet_bootstrap.cc:654] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:55.838243 3926 log.cc:824] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:55.839299 3925 log.cc:824] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:55.839347 3924 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: No bootstrap required, opened a new log
I20260929 16:39:55.839757 3924 ts_tablet_manager.cc:1397] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 16:39:55.841935 3925 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: No bootstrap required, opened a new log
I20260929 16:39:55.842185 3925 ts_tablet_manager.cc:1397] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 16:39:55.850889 3925 raft_consensus.cc:348] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.851405 3925 raft_consensus.cc:374] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:55.851516 3925 raft_consensus.cc:729] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a73416cb6a436c85fb52c0990352f5, State: Initialized, Role: FOLLOWER
I20260929 16:39:55.851173 3924 raft_consensus.cc:348] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.851657 3924 raft_consensus.cc:374] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:55.851740 3924 raft_consensus.cc:729] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88fcba94b36443d897685286fddde31, State: Initialized, Role: FOLLOWER
I20260929 16:39:55.852290 3925 consensus_queue.cc:260] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.853459 3924 consensus_queue.cc:260] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.855340 3926 tablet_bootstrap.cc:492] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: No bootstrap required, opened a new log
I20260929 16:39:55.855685 3926 ts_tablet_manager.cc:1397] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.014s
I20260929 16:39:55.857489 3906 heartbeater.cc:503] Master 127.17.205.62:46527 was elected leader, sending a full tablet report...
I20260929 16:39:55.857582 3925 ts_tablet_manager.cc:1428] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 16:39:55.860503 3924 ts_tablet_manager.cc:1428] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Time spent starting tablet: real 0.021s user 0.012s sys 0.004s
I20260929 16:39:55.860858 3776 heartbeater.cc:503] Master 127.17.205.62:46527 was elected leader, sending a full tablet report...
I20260929 16:39:55.865926 3926 raft_consensus.cc:348] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.866331 3926 raft_consensus.cc:374] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:55.866431 3926 raft_consensus.cc:729] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f783d2ddf6a4677800ce363002de8d5, State: Initialized, Role: FOLLOWER
I20260929 16:39:55.866978 3926 consensus_queue.cc:260] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.868011 3645 heartbeater.cc:503] Master 127.17.205.62:46527 was elected leader, sending a full tablet report...
I20260929 16:39:55.868497 3926 ts_tablet_manager.cc:1428] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:39:55.935802 3930 raft_consensus.cc:482] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:39:55.936148 3930 raft_consensus.cc:504] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.937927 3930 leader_election.cc:288] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f783d2ddf6a4677800ce363002de8d5 (127.17.205.1:44177), b88fcba94b36443d897685286fddde31 (127.17.205.2:38759)
I20260929 16:39:55.943539 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.938182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.17.205.3:42813)
0929 16:39:55.938379 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.938400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.938535 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.939040 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.939046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.939074 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:55.939264 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.939276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.940253 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.940258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.941667 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.941679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.942786 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.942792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.942968 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.942992 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.943070 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.943240 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:39:55.943532 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.938380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:42813 (local address 127.17.205.1:44177)
0929 16:39:55.938608 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.938614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.938634 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:39:55.938712 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.938717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.938816 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.938909 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:55.939411 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.940120 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.941830 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.942628 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.942802 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.942877 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.943119 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.943283 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:39:55.944244 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.938851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:32911 (local address 127.17.205.2:38759)
0929 16:39:55.939043 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.939048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.939234 (+ 186us) server_negotiation.cc:407] Connection header received
0929 16:39:55.939413 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.939416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.939473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.939543 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:55.940028 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.940712 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.942517 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.943360 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.943496 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.943560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.943925 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.944028 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:39:55.944890 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.938510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38759 (local address 127.17.205.3:32911)
0929 16:39:55.939123 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:55.939136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:55.939278 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:55.939567 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:55.939572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:55.939594 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:55.939889 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.939897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.940869 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.940874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.942340 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:55.942352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.943512 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.943518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:55.943701 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.943735 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:39:55.944441 (+ 706us) client_negotiation.cc:240] Negotiation successful
0929 16:39:55.944575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":1,"thread_start_us":302,"threads_started":1}
I20260929 16:39:55.944695 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cfaf6f1d5b4bc5ac03e90447e67e12" candidate_uuid: "15a73416cb6a436c85fb52c0990352f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f783d2ddf6a4677800ce363002de8d5" is_pre_election: true
I20260929 16:39:55.945266 3600 raft_consensus.cc:2454] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a73416cb6a436c85fb52c0990352f5 in term 0.
I20260929 16:39:55.945919 3797 leader_election.cc:302] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a73416cb6a436c85fb52c0990352f5, 1f783d2ddf6a4677800ce363002de8d5; no voters:
I20260929 16:39:55.945863 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cfaf6f1d5b4bc5ac03e90447e67e12" candidate_uuid: "15a73416cb6a436c85fb52c0990352f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88fcba94b36443d897685286fddde31" is_pre_election: true
I20260929 16:39:55.946297 3930 raft_consensus.cc:2781] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:39:55.946296 3731 raft_consensus.cc:2454] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a73416cb6a436c85fb52c0990352f5 in term 0.
I20260929 16:39:55.946411 3930 raft_consensus.cc:482] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:39:55.946506 3930 raft_consensus.cc:3037] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:55.948992 3930 raft_consensus.cc:504] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.949646 3930 leader_election.cc:288] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [CANDIDATE]: Term 1 election: Requested vote from peers 1f783d2ddf6a4677800ce363002de8d5 (127.17.205.1:44177), b88fcba94b36443d897685286fddde31 (127.17.205.2:38759)
I20260929 16:39:55.950146 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cfaf6f1d5b4bc5ac03e90447e67e12" candidate_uuid: "15a73416cb6a436c85fb52c0990352f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f783d2ddf6a4677800ce363002de8d5"
I20260929 16:39:55.950184 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cfaf6f1d5b4bc5ac03e90447e67e12" candidate_uuid: "15a73416cb6a436c85fb52c0990352f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88fcba94b36443d897685286fddde31"
I20260929 16:39:55.950330 3600 raft_consensus.cc:3037] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:55.950380 3731 raft_consensus.cc:3037] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:55.952729 3600 raft_consensus.cc:2454] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a73416cb6a436c85fb52c0990352f5 in term 1.
I20260929 16:39:55.952788 3731 raft_consensus.cc:2454] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a73416cb6a436c85fb52c0990352f5 in term 1.
I20260929 16:39:55.953172 3797 leader_election.cc:302] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a73416cb6a436c85fb52c0990352f5, 1f783d2ddf6a4677800ce363002de8d5; no voters:
I20260929 16:39:55.953439 3930 raft_consensus.cc:2781] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:55.953874 3930 raft_consensus.cc:686] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 1 LEADER]: Becoming Leader. State: Replica: 15a73416cb6a436c85fb52c0990352f5, State: Running, Role: LEADER
I20260929 16:39:55.954421 3930 consensus_queue.cc:237] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } }
I20260929 16:39:55.959139 3459 catalog_manager.cc:5495] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a73416cb6a436c85fb52c0990352f5 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "15a73416cb6a436c85fb52c0990352f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a73416cb6a436c85fb52c0990352f5" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20260929 16:39:55.978516 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:55.973512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.17.205.3:43055)
0929 16:39:55.973737 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:55.973742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:55.973760 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:39:55.973854 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:55.973858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:55.973957 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:55.974095 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:39:55.974437 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.975185 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.976588 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:55.976986 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:55.977072 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:55.977498 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:39:55.978037 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:39:55.978148 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:55.978188 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 16:39:55.978256 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 16:39:55.981389 3907 tablet.cc:2369] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:39:55.987637 3731 raft_consensus.cc:1261] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 1 FOLLOWER]: Refusing update from remote peer 15a73416cb6a436c85fb52c0990352f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:55.987635 3600 raft_consensus.cc:1261] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 1 FOLLOWER]: Refusing update from remote peer 15a73416cb6a436c85fb52c0990352f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:39:55.988704 3934 consensus_queue.cc:1035] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f783d2ddf6a4677800ce363002de8d5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 44177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:55.989073 3930 consensus_queue.cc:1035] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88fcba94b36443d897685286fddde31" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:39:56.001044 3937 mvcc.cc:205] Tried to move back new op lower bound from 7334707183554383872 to 7334707183433564160. Current Snapshot: MvccSnapshot[applied={T|T < 7334707183554383872}]
I20260929 16:39:56.015056 3457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:56.015439 3457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:39:56.019506 3457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 372073a525184fe7b73870334d509937: Sending DeleteTablet for 3 replicas of tablet 68cfaf6f1d5b4bc5ac03e90447e67e12
I20260929 16:39:56.020728 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:56 UTC) from {username='slave'} at 127.0.0.1:56574
I20260929 16:39:56.020790 3711 tablet_service.cc:1514] Processing DeleteTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:56 UTC) from {username='slave'} at 127.0.0.1:60988
I20260929 16:39:56.021863 3580 tablet_service.cc:1514] Processing DeleteTablet for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:39:56 UTC) from {username='slave'} at 127.0.0.1:43402
I20260929 16:39:56.022642 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3518
I20260929 16:39:56.034334 3946 tablet_replica.cc:330] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: stopping tablet replica
I20260929 16:39:56.035362 3948 tablet_replica.cc:330] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: stopping tablet replica
I20260929 16:39:56.035899 3948 raft_consensus.cc:2229] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:56.036384 3946 raft_consensus.cc:2229] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:39:56.036590 3948 raft_consensus.cc:2258] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:56.037313 3946 raft_consensus.cc:2258] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:56.039072 3948 ts_tablet_manager.cc:1905] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 1f783d2ddf6a4677800ce363002de8d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:56.041838 3947 tablet_replica.cc:330] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: stopping tablet replica
I20260929 16:39:56.042340 3947 raft_consensus.cc:2229] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:39:56.043322 3947 raft_consensus.cc:2258] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:39:56.043532 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.024188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.17.205.1:44177)
0929 16:39:56.024388 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:56.024395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:56.034553 (+ 10158us) server_negotiation.cc:407] Connection header received
0929 16:39:56.034784 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:56.034792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:56.034931 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:56.035065 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:56.036075 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.037355 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.039371 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.040030 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.040134 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:56.041645 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:56.041656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:56.041675 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:56.042515 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:56.042585 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:56.042612 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:56.042751 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:56.043019 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:39:56.043143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:39:56.044492 3946 ts_tablet_manager.cc:1905] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:56.044759 3640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:56.046599 3947 ts_tablet_manager.cc:1905] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:39:56.053781 3947 ts_tablet_manager.cc:1918] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:56.053834 3946 ts_tablet_manager.cc:1918] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:39:56.054028 3947 log.cc:1196] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/68cfaf6f1d5b4bc5ac03e90447e67e12
I20260929 16:39:56.054209 3946 log.cc:1196] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/68cfaf6f1d5b4bc5ac03e90447e67e12
I20260929 16:39:56.054601 3947 ts_tablet_manager.cc:1939] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P b88fcba94b36443d897685286fddde31: Deleting consensus metadata
I20260929 16:39:56.054893 3946 ts_tablet_manager.cc:1939] T 68cfaf6f1d5b4bc5ac03e90447e67e12 P 15a73416cb6a436c85fb52c0990352f5: Deleting consensus metadata
I20260929 16:39:56.056814 3443 catalog_manager.cc:4841] TS 15a73416cb6a436c85fb52c0990352f5 (127.17.205.3:43055): tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 (table test-table [id=2482f8db343f4807a07eb8e8f6bfdedc]) successfully deleted
I20260929 16:39:56.056910 3446 catalog_manager.cc:4841] TS b88fcba94b36443d897685286fddde31 (127.17.205.2:38759): tablet 68cfaf6f1d5b4bc5ac03e90447e67e12 (table test-table [id=2482f8db343f4807a07eb8e8f6bfdedc]) successfully deleted
I20260929 16:39:56.147596 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3518
I20260929 16:39:56.168320 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3649
W20260929 16:39:56.168314 3446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:44177 (error 108)
I20260929 16:39:56.170420 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.169802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43424)
0929 16:39:56.170073 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.170227 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260929 16:39:56.170737 3446 catalog_manager.cc:4584] TS 1f783d2ddf6a4677800ce363002de8d5 (127.17.205.1:44177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68cfaf6f1d5b4bc5ac03e90447e67e12: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
I20260929 16:39:56.176182 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.169567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.17.205.2:38759)
0929 16:39:56.169747 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:56.169753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:56.169850 (+ 97us) server_negotiation.cc:407] Connection header received
0929 16:39:56.170038 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:56.170042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:56.170145 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:56.170229 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:56.170697 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.171570 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.172856 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.173280 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.173398 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:56.174488 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:56.174497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:56.174513 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:56.175430 (+ 917us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:56.175498 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:56.175522 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:56.175624 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:56.175758 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 16:39:56.175875 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:39:56.176954 3771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:56.224653 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.224045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43428)
0929 16:39:56.224302 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.224440 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:39:56.225905 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.225487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43432)
0929 16:39:56.225647 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.225739 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:39:56.281279 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3649
I20260929 16:39:56.286818 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.286239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43434)
0929 16:39:56.286486 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.286601 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:39:56.288089 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.287652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43442)
0929 16:39:56.287826 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.287928 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:39:56.296371 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3779
I20260929 16:39:56.304793 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.297171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.17.205.3:43055)
0929 16:39:56.297353 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:56.297357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:56.297423 (+ 66us) server_negotiation.cc:407] Connection header received
0929 16:39:56.297604 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:56.297608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:56.297716 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:56.297806 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:56.298331 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.299148 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.301220 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:56.301727 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:56.301856 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:56.303155 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:56.303165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:56.303182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:39:56.303899 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:56.303957 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:39:56.303980 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:56.304090 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:56.304255 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 16:39:56.304363 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:39:56.305572 3901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:56.368614 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.368022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43444)
0929 16:39:56.368279 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.368410 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:39:56.369751 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.369289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43448)
0929 16:39:56.369501 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.369598 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 16:39:56.430462 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3779
I20260929 16:39:56.446866 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3428
I20260929 16:39:56.448025 3489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:39:56.558555 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:56.558115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:44177 (local address 127.0.0.1:43456)
0929 16:39:56.558315 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:56.558398 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:39:56.558539 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3428
2026-09-29T16:39:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2741 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:39:56 dist-test-slave-3h01 krb5kdc[3957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:39:56 dist-test-slave-3h01 krb5kdc[3957](info): set up 2 sockets
Sep 29 16:39:56 dist-test-slave-3h01 krb5kdc[3957](info): commencing operation
krb5kdc: starting...
W20260929 16:39:57.567453 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:39:57 dist-test-slave-3h01 krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:39:57.806533 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41063
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:58.131345 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:58.131649 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:58.131744 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:58.138875 3970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:39:58.138993 3970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:58.139038 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:58.139097 3970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:39:58.139139 3970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:39:58.149947 3970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:41063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:58.151268 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:58.152546 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:58.164381 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:58.164472 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:58.164459 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:58.165283 3970 server_base.cc:1034] running on GCE node
I20260929 16:39:58.166075 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:58.167589 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:58.168826 3970 hybrid_clock.cc:648] HybridClock initialized: now 1790699998168761 us; error 53 us; skew 500 ppm
I20260929 16:39:58.169169 3970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:58 dist-test-slave-3h01 krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:58.172135 3970 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:39:58.173518 3970 webserver.cc:460] Webserver started at http://127.17.205.62:43223/ using document root <none> and password file <none>
I20260929 16:39:58.174196 3970 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:58.174307 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:58.174592 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:58.177606 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "33a2439122b8466e87aa3fa4b1a9ca72"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "58683f085d66fcfb611142d3b4b7ee9b"
server_key_iv: "cc330b63dab3925a48ce0b4c2886eff2"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.178408 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "33a2439122b8466e87aa3fa4b1a9ca72"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "58683f085d66fcfb611142d3b4b7ee9b"
server_key_iv: "cc330b63dab3925a48ce0b4c2886eff2"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.183669 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:39:58.187047 3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:58.188431 3970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:39:58.188583 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "33a2439122b8466e87aa3fa4b1a9ca72"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "58683f085d66fcfb611142d3b4b7ee9b"
server_key_iv: "cc330b63dab3925a48ce0b4c2886eff2"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.188771 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:58.213351 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:58.217505 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:58.217794 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:58.234522 3970 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:41063
I20260929 16:39:58.234545 4036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:41063 every 8 connection(s)
I20260929 16:39:58.235918 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:39:58.239035 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 3970
I20260929 16:39:58.239202 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:39:58.239671 18228 external_mini_cluster.cc:1428] Setting key 72421522774cd6d14b3b68f99e9dc4b1
I20260929 16:39:58.239825 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:58.253753 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: Bootstrap starting.
Sep 29 16:39:58 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:58.257438 4037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:58.258816 4037 log.cc:824] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:58.262380 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: No bootstrap required, opened a new log
I20260929 16:39:58.263891 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:58.241563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.17.205.62:41063)
0929 16:39:58.242547 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:58.242563 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:58.242877 (+ 314us) server_negotiation.cc:407] Connection header received
0929 16:39:58.244716 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:58.244749 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:58.245301 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:58.245777 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:58.247105 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.249013 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.250332 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.251437 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.251623 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:58.255466 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:58.255477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:58.255497 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:58.255543 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:58.259266 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:58.260152 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:58.260158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:58.260167 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:58.260336 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:58.260722 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:58.260726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:58.260729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:58.261405 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:58.261697 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:58.262059 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:39:58.262504 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":377,"threads_started":1}
I20260929 16:39:58.270710 4037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } }
I20260929 16:39:58.271052 4037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:58.271127 4037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a2439122b8466e87aa3fa4b1a9ca72, State: Initialized, Role: FOLLOWER
I20260929 16:39:58.271787 4037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } }
I20260929 16:39:58.272048 4037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:39:58.272190 4037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:39:58.272390 4037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:39:58.274483 4037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } }
I20260929 16:39:58.274950 4037 leader_election.cc:302] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a2439122b8466e87aa3fa4b1a9ca72; no voters:
I20260929 16:39:58.275503 4037 leader_election.cc:288] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:39:58.275633 4042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:39:58.276628 4042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [term 1 LEADER]: Becoming Leader. State: Replica: 33a2439122b8466e87aa3fa4b1a9ca72, State: Running, Role: LEADER
I20260929 16:39:58.277166 4042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } }
I20260929 16:39:58.277678 4037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:39:58.279811 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a2439122b8466e87aa3fa4b1a9ca72. Latest consensus state: current_term: 1 leader_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } } }
I20260929 16:39:58.279837 4043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2439122b8466e87aa3fa4b1a9ca72" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41063 } } }
I20260929 16:39:58.280298 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:58.280385 4043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72 [sys.catalog]: This master's current role is: LEADER
I20260929 16:39:58.282147 4049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:39:58.286748 4049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:39:58.295120 4049 catalog_manager.cc:1347] Generated new cluster ID: 73f46fb5cf49404385afb17af1354b3b
I20260929 16:39:58.295298 4049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:39:58.313509 4049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:39:58.315173 4049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:39:58.324435 4049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: Generated new TSK 0
I20260929 16:39:58.325322 4049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:39:58.384384 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:58.707965 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:58.708245 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:58.708338 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:58.715474 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:58.715584 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:58.715706 4063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:39:58.725692 4063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:58.726951 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:58.728386 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:58.741210 4068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:58.741183 4069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:58.741240 4071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:58.741932 4063 server_base.cc:1034] running on GCE node
I20260929 16:39:58.742566 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:58.743578 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:58.744777 4063 hybrid_clock.cc:648] HybridClock initialized: now 1790699998744715 us; error 48 us; skew 500 ppm
I20260929 16:39:58.745102 4063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:58 dist-test-slave-3h01 krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:58.748199 4063 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:39:58.749643 4063 webserver.cc:460] Webserver started at http://127.17.205.1:42621/ using document root <none> and password file <none>
I20260929 16:39:58.750355 4063 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:58.750468 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:58.750770 4063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:58.753762 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "0feab61ae1aa5635faedff10cd31e0cf"
server_key_iv: "e720a07782e352c264dfbc9c94736a22"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.754540 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "0feab61ae1aa5635faedff10cd31e0cf"
server_key_iv: "e720a07782e352c264dfbc9c94736a22"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.759905 4063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 16:39:58.763540 4078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:58.764880 4063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:39:58.765041 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5"
format_stamp: "Formatted at 2026-09-29 16:39:58 on dist-test-slave-3h01"
server_key: "0feab61ae1aa5635faedff10cd31e0cf"
server_key_iv: "e720a07782e352c264dfbc9c94736a22"
server_key_version: "encryptionkey@0"
I20260929 16:39:58.765226 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:58.804284 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:58.808287 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:58.808564 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:58.809520 4063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:58.810839 4063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:58.810925 4063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:58.811049 4063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:58.811102 4063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:58.841367 4063 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:40843
I20260929 16:39:58.841444 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:40843 every 8 connection(s)
I20260929 16:39:58.842727 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:39:58.847529 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4063
I20260929 16:39:58.847693 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:39:58.848093 18228 external_mini_cluster.cc:1428] Setting key 25c09c30cb807c1fd0c7d53ae71bcae5
Sep 29 16:39:58 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699998, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:58.862499 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:58.845254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:46389 (local address 127.17.205.62:41063)
0929 16:39:58.845675 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:58.845680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:58.846618 (+ 938us) server_negotiation.cc:407] Connection header received
0929 16:39:58.848615 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:58.848621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:58.848708 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:58.848804 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:58.850612 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.851376 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.852932 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.853340 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.853456 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:58.857040 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:58.857045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:58.857048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:58.857083 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:58.859462 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:58.860204 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:58.860214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:58.860217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:58.860341 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:58.860751 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:58.860756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:58.860758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:58.861026 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:58.861156 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:58.861887 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 16:39:58.862123 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":221,"threads_started":1}
I20260929 16:39:58.863617 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:58.845701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41063 (local address 127.17.205.1:46389)
0929 16:39:58.846438 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:58.846485 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:58.848329 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:58.849107 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:58.849121 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:58.849823 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:58.850434 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:58.850452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.851515 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.851518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:58.852792 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:58.852805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:58.853570 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:58.853578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:58.853825 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:58.854469 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:58.854493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:58.856802 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:58.859606 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:58.859617 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:58.859634 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:58.860086 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:58.860441 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:58.860446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:58.860450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:58.860644 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:58.861155 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:58.861163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:58.861514 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 16:39:58.861773 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 16:39:58.862345 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":298,"threads_started":1}
I20260929 16:39:58.865235 4191 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41063
I20260929 16:39:58.865545 4191 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:58.866307 4191 heartbeater.cc:511] Master 127.17.205.62:41063 requested a full tablet report, sending...
I20260929 16:39:58.868824 4002 ts_manager.cc:194] Registered new tserver with Master: 26f13c6bc4d44b8f9e48c5afe9947fe5 (127.17.205.1:40843)
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:39:58.870839 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:46389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:39:58.898399 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:59.223321 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:59.223604 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:59.223762 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:59.230995 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:59.231103 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:59.231271 4197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:39:59.240962 4197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:59.242401 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:59.243791 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:59.256167 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:59.256240 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:59.256210 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:59.257018 4197 server_base.cc:1034] running on GCE node
I20260929 16:39:59.257613 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:59.258539 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:59.259743 4197 hybrid_clock.cc:648] HybridClock initialized: now 1790699999259682 us; error 51 us; skew 500 ppm
I20260929 16:39:59.260061 4197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:59 dist-test-slave-3h01 krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699999, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:59.263051 4197 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:39:59.264424 4197 webserver.cc:460] Webserver started at http://127.17.205.2:40637/ using document root <none> and password file <none>
I20260929 16:39:59.265074 4197 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:59.265187 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:59.265488 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:59.268400 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "e81d6a385fe34e28ae4e45bfb99ab41d"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "ecca004c4c55b68dfb66073849738f87"
server_key_iv: "87abd9a93c28a4725e731fc3a5c4e5cf"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.269135 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "e81d6a385fe34e28ae4e45bfb99ab41d"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "ecca004c4c55b68dfb66073849738f87"
server_key_iv: "87abd9a93c28a4725e731fc3a5c4e5cf"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.274062 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:39:59.277374 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.278739 4197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:39:59.278882 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "e81d6a385fe34e28ae4e45bfb99ab41d"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "ecca004c4c55b68dfb66073849738f87"
server_key_iv: "87abd9a93c28a4725e731fc3a5c4e5cf"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.279067 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:59.298012 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:59.301952 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:59.302227 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:59.303151 4197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:59.304471 4197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:59.304545 4197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.304668 4197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:59.304720 4197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.333784 4197 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:46119
I20260929 16:39:59.333827 4324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:46119 every 8 connection(s)
I20260929 16:39:59.335037 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:39:59.341082 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4197
I20260929 16:39:59.341274 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:39:59.341670 18228 external_mini_cluster.cc:1428] Setting key c6e02a66667f9ca7d14c2d126359a5ad
Sep 29 16:39:59 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699999, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:59.355901 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.337530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:47759 (local address 127.17.205.62:41063)
0929 16:39:59.337715 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.337720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.338802 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 16:39:59.340605 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.340612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.340700 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.340786 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:59.342851 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.343774 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.345368 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.345958 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.346080 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.349973 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:59.349980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:59.349984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:59.350023 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:59.352849 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.353730 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.353734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.353736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.353865 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.354281 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.354284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.354286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.354538 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:59.354642 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.355468 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.355645 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:39:59.357278 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.337960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41063 (local address 127.17.205.2:47759)
0929 16:39:59.338633 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:59.338668 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:59.340329 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:59.341468 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:59.341476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:59.342008 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:59.342690 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.342711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.343918 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.343922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.345214 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.345228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.346146 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.346150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.346353 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.347034 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:59.347067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:59.349720 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:59.352982 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:59.352991 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:59.353010 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:59.353575 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:59.353962 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:59.353966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:59.353969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:59.354193 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:59.354648 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:59.354657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:59.355087 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 16:39:59.355399 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 16:39:59.355938 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260929 16:39:59.358714 4325 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41063
I20260929 16:39:59.359047 4325 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:59.359913 4325 heartbeater.cc:511] Master 127.17.205.62:41063 requested a full tablet report, sending...
I20260929 16:39:59.361529 4002 ts_manager.cc:194] Registered new tserver with Master: e81d6a385fe34e28ae4e45bfb99ab41d (127.17.205.2:46119)
I20260929 16:39:59.362363 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:47759
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:39:59.394469 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:39:59.714082 4330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:39:59.714372 4330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:39:59.714464 4330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:39:59.721638 4330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:39:59.721760 4330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:39:59.721884 4330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:39:59.731753 4330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:41063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:39:59.733040 4330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:39:59.734292 4330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:39:59.746898 4335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:59.746845 4338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:39:59.746912 4336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:39:59.747774 4330 server_base.cc:1034] running on GCE node
I20260929 16:39:59.748370 4330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:39:59.749321 4330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:39:59.750519 4330 hybrid_clock.cc:648] HybridClock initialized: now 1790699999750468 us; error 48 us; skew 500 ppm
I20260929 16:39:59.750828 4330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:39:59 dist-test-slave-3h01 krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699999, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:39:59.753892 4330 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:39:59.755357 4330 webserver.cc:460] Webserver started at http://127.17.205.3:33283/ using document root <none> and password file <none>
I20260929 16:39:59.756001 4330 fs_manager.cc:359] Metadata directory not provided
I20260929 16:39:59.756098 4330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:39:59.756366 4330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:39:59.759106 4330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "c8985825c32a468188399f8cf41f54cf"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "f7804ff8f90dd6131cd1bfd85e149fef"
server_key_iv: "cc10795218281a733ce0b1e912d61102"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.759836 4330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "c8985825c32a468188399f8cf41f54cf"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "f7804ff8f90dd6131cd1bfd85e149fef"
server_key_iv: "cc10795218281a733ce0b1e912d61102"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.764911 4330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:39:59.768303 4345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.769855 4330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:39:59.770011 4330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "c8985825c32a468188399f8cf41f54cf"
format_stamp: "Formatted at 2026-09-29 16:39:59 on dist-test-slave-3h01"
server_key: "f7804ff8f90dd6131cd1bfd85e149fef"
server_key_iv: "cc10795218281a733ce0b1e912d61102"
server_key_version: "encryptionkey@0"
I20260929 16:39:59.770190 4330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:39:59.793642 4330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:39:59.797874 4330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:39:59.798133 4330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:39:59.799064 4330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:39:59.800421 4330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:39:59.800493 4330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.800580 4330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:39:59.800621 4330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:39:59.829227 4330 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:32837
I20260929 16:39:59.829281 4457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:32837 every 8 connection(s)
I20260929 16:39:59.830598 4330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:39:59.836366 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4330
I20260929 16:39:59.836570 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:39:59.837016 18228 external_mini_cluster.cc:1428] Setting key ddaa65d2d327fc3936fb95f2743eb5c5
Sep 29 16:39:59 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699999, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:39:59.849706 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.833090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:51147 (local address 127.17.205.62:41063)
0929 16:39:59.833262 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.833267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.834404 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 16:39:59.836251 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.836258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.836339 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.836426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:59.838104 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.839035 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.840654 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.841112 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.841222 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.844702 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:59.844708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:59.844712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:59.844765 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:59.847274 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.847912 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.847915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.847917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.848036 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.848349 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.848352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.848354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.848596 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:59.848699 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.849308 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.849466 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:39:59.850665 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.833550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:41063 (local address 127.17.205.3:51147)
0929 16:39:59.834227 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:59.834268 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:59.835969 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:59.836691 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:59.836700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:59.837249 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:39:59.837919 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.837939 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.839167 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.839171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.840523 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.840535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.841336 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.841344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.841557 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.842185 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:39:59.842210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:39:59.844476 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:39:59.847396 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:59.847403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:59.847415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:59.847823 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:59.848109 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:39:59.848112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:39:59.848114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:39:59.848274 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:39:59.848699 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:39:59.848705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:39:59.849016 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 16:39:59.849225 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:39:59.849596 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":256,"threads_started":1}
I20260929 16:39:59.851922 4458 heartbeater.cc:347] Connected to a master server at 127.17.205.62:41063
I20260929 16:39:59.852227 4458 heartbeater.cc:464] Registering TS with master...
I20260929 16:39:59.852916 4458 heartbeater.cc:511] Master 127.17.205.62:41063 requested a full tablet report, sending...
I20260929 16:39:59.854249 4001 ts_manager.cc:194] Registered new tserver with Master: c8985825c32a468188399f8cf41f54cf (127.17.205.3:32837)
I20260929 16:39:59.855065 4001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:51147
I20260929 16:39:59.864391 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:39:59.874195 4191 heartbeater.cc:503] Master 127.17.205.62:41063 was elected leader, sending a full tablet report...
I20260929 16:39:59.881450 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.868416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.17.205.62:41063)
0929 16:39:59.868570 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.868576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.869059 (+ 483us) server_negotiation.cc:407] Connection header received
0929 16:39:59.869218 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.869224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.869291 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.869371 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:39:59.870798 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.871578 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.872932 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.873361 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.873494 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.876249 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:39:59.876254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:39:59.876258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:39:59.876288 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:39:59.878697 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.879496 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.879500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.879503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.879635 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:39:59.880072 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:39:59.880075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:39:59.880077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:39:59.880395 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:39:59.880645 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.880998 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.881184 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:39:59.885452 4001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:39:59.892347 4001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:39:59.915707 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.903945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:32837 (local address 127.0.0.1:38680)
0929 16:39:59.904935 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:59.904973 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:59.905141 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:59.907471 (+ 2330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:59.907478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:59.907509 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:59.907948 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.907964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.910051 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.910056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.912418 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.912434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.914676 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.914682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.914922 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.914977 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:39:59.915111 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:39:59.915331 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":2747,"threads_started":1}
I20260929 16:39:59.916421 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.904208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.17.205.3:32837)
0929 16:39:59.906640 (+ 2432us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.906646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.906670 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:39:59.906744 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.906749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.906946 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.907126 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:59.908163 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.909877 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.912604 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.914147 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.914274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.914528 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.915948 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.916113 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2302,"thread_start_us":2167,"threads_started":1}
I20260929 16:39:59.921660 4393 tablet_service.cc:1467] Processing CreateTablet for tablet 90b175d1d02140418d899300d8fb546b (DEFAULT_TABLE table=test-table [id=112b8e48808945208578cddfb5085d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:59.923249 4393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b175d1d02140418d899300d8fb546b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:59.924414 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.904271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:46119 (local address 127.0.0.1:49436)
0929 16:39:59.906948 (+ 2677us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:59.906966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:59.907077 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:59.908040 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:59.916889 (+ 8849us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:59.916954 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:59.917584 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.917600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.919910 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.919916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.921785 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.921801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.923675 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.923682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.923842 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.923876 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:39:59.923957 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:39:59.924084 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"thread_start_us":1207,"threads_started":1}
I20260929 16:39:59.924455 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.904884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.17.205.2:46119)
0929 16:39:59.906263 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.906271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.907196 (+ 925us) server_negotiation.cc:407] Connection header received
0929 16:39:59.907346 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.907355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.907606 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.907805 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:59.917860 (+ 10055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.919741 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.921966 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.923458 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.923587 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.923797 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.924009 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.924161 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1145,"threads_started":1}
I20260929 16:39:59.928344 4260 tablet_service.cc:1467] Processing CreateTablet for tablet 90b175d1d02140418d899300d8fb546b (DEFAULT_TABLE table=test-table [id=112b8e48808945208578cddfb5085d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:59.929474 4260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b175d1d02140418d899300d8fb546b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:59.937381 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.907060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56536)
0929 16:39:59.916714 (+ 9654us) negotiation.cc:107] Waiting for socket to connect
0929 16:39:59.916731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:39:59.916852 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:39:59.917802 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:39:59.917807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:39:59.917833 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:39:59.918032 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.918042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.932788 (+ 14746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.932795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.934667 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:39:59.934686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.936595 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.936602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:39:59.936784 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.936821 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:39:59.936936 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:39:59.937080 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9545,"thread_start_us":10334,"threads_started":1}
I20260929 16:39:59.937563 4476 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Bootstrap starting.
I20260929 16:39:59.938325 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:39:59.907414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.17.205.1:40843)
0929 16:39:59.916033 (+ 8619us) server_negotiation.cc:206] Beginning negotiation
0929 16:39:59.916041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:39:59.916800 (+ 759us) server_negotiation.cc:407] Connection header received
0929 16:39:59.916989 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:39:59.916997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:39:59.917249 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:39:59.918217 (+ 968us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:39:59.918444 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.932577 (+ 14133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.934903 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:39:59.936403 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:39:59.936561 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:39:59.936847 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 16:39:59.937817 (+ 970us) server_negotiation.cc:299] Negotiation successful
0929 16:39:59.937985 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8439,"thread_start_us":9180,"threads_started":1}
I20260929 16:39:59.941427 4476 tablet_bootstrap.cc:654] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:59.942660 4476 log.cc:824] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:59.942203 4126 tablet_service.cc:1467] Processing CreateTablet for tablet 90b175d1d02140418d899300d8fb546b (DEFAULT_TABLE table=test-table [id=112b8e48808945208578cddfb5085d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:39:59.942903 4477 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Bootstrap starting.
I20260929 16:39:59.943907 4126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b175d1d02140418d899300d8fb546b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:39:59.947306 4477 tablet_bootstrap.cc:654] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:59.948523 4477 log.cc:824] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:59.948704 4476 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: No bootstrap required, opened a new log
I20260929 16:39:59.948971 4476 ts_tablet_manager.cc:1397] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 16:39:59.951601 4477 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: No bootstrap required, opened a new log
I20260929 16:39:59.951882 4477 ts_tablet_manager.cc:1397] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260929 16:39:59.956804 4476 raft_consensus.cc:348] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.957139 4476 raft_consensus.cc:374] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:59.957247 4476 raft_consensus.cc:729] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8985825c32a468188399f8cf41f54cf, State: Initialized, Role: FOLLOWER
I20260929 16:39:59.957847 4476 consensus_queue.cc:260] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.958309 4480 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Bootstrap starting.
I20260929 16:39:59.959288 4458 heartbeater.cc:503] Master 127.17.205.62:41063 was elected leader, sending a full tablet report...
I20260929 16:39:59.960332 4476 ts_tablet_manager.cc:1428] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260929 16:39:59.961656 4480 tablet_bootstrap.cc:654] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Neither blocks nor log segments found. Creating new log.
I20260929 16:39:59.961704 4477 raft_consensus.cc:348] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.962124 4477 raft_consensus.cc:374] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:59.962226 4477 raft_consensus.cc:729] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81d6a385fe34e28ae4e45bfb99ab41d, State: Initialized, Role: FOLLOWER
I20260929 16:39:59.962779 4480 log.cc:824] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:39:59.963039 4477 consensus_queue.cc:260] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.964187 4325 heartbeater.cc:503] Master 127.17.205.62:41063 was elected leader, sending a full tablet report...
I20260929 16:39:59.964905 4477 ts_tablet_manager.cc:1428] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:39:59.965858 4480 tablet_bootstrap.cc:492] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: No bootstrap required, opened a new log
I20260929 16:39:59.966092 4480 ts_tablet_manager.cc:1397] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 16:39:59.973454 4480 raft_consensus.cc:348] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.973834 4480 raft_consensus.cc:374] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:39:59.973927 4480 raft_consensus.cc:729] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f13c6bc4d44b8f9e48c5afe9947fe5, State: Initialized, Role: FOLLOWER
I20260929 16:39:59.974498 4480 consensus_queue.cc:260] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:39:59.976259 4480 ts_tablet_manager.cc:1428] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
W20260929 16:40:00.082167 4459 tablet.cc:2369] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:00.087010 4326 tablet.cc:2369] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:00.095042 4192 tablet.cc:2369] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:00.098062 4484 raft_consensus.cc:482] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:40:00.098343 4484 raft_consensus.cc:504] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:40:00.100059 4484 leader_election.cc:288] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8985825c32a468188399f8cf41f54cf (127.17.205.3:32837), e81d6a385fe34e28ae4e45bfb99ab41d (127.17.205.2:46119)
I20260929 16:40:00.108197 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.100544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:51655 (local address 127.17.205.3:32837)
0929 16:40:00.100754 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.100760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.101425 (+ 665us) server_negotiation.cc:407] Connection header received
0929 16:40:00.101561 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.101566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.101636 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.101720 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:00.102279 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.103371 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.105856 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.107051 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.107205 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.107309 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.107830 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.107962 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:40:00.108197 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.100635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:36055 (local address 127.17.205.2:46119)
0929 16:40:00.100860 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.100866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.101199 (+ 333us) server_negotiation.cc:407] Connection header received
0929 16:40:00.101403 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.101409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.101488 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.101576 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:00.102145 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.103267 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.105856 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.107054 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.107205 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.107309 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.107784 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.107933 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:40:00.108323 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.100424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:46119 (local address 127.17.205.1:36055)
0929 16:40:00.101088 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.101104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:00.101270 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:00.101589 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:00.101592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:00.101610 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:00.101992 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:00.102005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.103388 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.103392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:00.105684 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:00.105702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.107335 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.107341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:00.107556 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.107612 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:40:00.107712 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:40:00.107861 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":32,"thread_start_us":344,"threads_started":1}
I20260929 16:40:00.108568 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.100464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:32837 (local address 127.17.205.1:51655)
0929 16:40:00.101331 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.101345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:00.101452 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:00.101763 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:00.101768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:00.101788 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:00.102146 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:00.102157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.103500 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.103505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:00.105690 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:00.105702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.107309 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.107319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:00.107556 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.107618 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:40:00.107732 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:40:00.107861 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":80,"thread_start_us":557,"threads_started":1}
I20260929 16:40:00.109422 4280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b175d1d02140418d899300d8fb546b" candidate_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" is_pre_election: true
I20260929 16:40:00.109575 4413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b175d1d02140418d899300d8fb546b" candidate_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8985825c32a468188399f8cf41f54cf" is_pre_election: true
I20260929 16:40:00.109967 4280 raft_consensus.cc:2454] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f13c6bc4d44b8f9e48c5afe9947fe5 in term 0.
I20260929 16:40:00.110025 4413 raft_consensus.cc:2454] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f13c6bc4d44b8f9e48c5afe9947fe5 in term 0.
I20260929 16:40:00.110674 4082 leader_election.cc:302] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f13c6bc4d44b8f9e48c5afe9947fe5, c8985825c32a468188399f8cf41f54cf; no voters:
I20260929 16:40:00.111093 4484 raft_consensus.cc:2781] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:40:00.111272 4484 raft_consensus.cc:482] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:40:00.111373 4484 raft_consensus.cc:3037] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:00.113533 4484 raft_consensus.cc:504] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:40:00.114104 4484 leader_election.cc:288] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [CANDIDATE]: Term 1 election: Requested vote from peers c8985825c32a468188399f8cf41f54cf (127.17.205.3:32837), e81d6a385fe34e28ae4e45bfb99ab41d (127.17.205.2:46119)
I20260929 16:40:00.114566 4413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b175d1d02140418d899300d8fb546b" candidate_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8985825c32a468188399f8cf41f54cf"
I20260929 16:40:00.114566 4280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b175d1d02140418d899300d8fb546b" candidate_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d"
I20260929 16:40:00.114794 4413 raft_consensus.cc:3037] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:00.114787 4280 raft_consensus.cc:3037] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:00.117916 4413 raft_consensus.cc:2454] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f13c6bc4d44b8f9e48c5afe9947fe5 in term 1.
I20260929 16:40:00.117918 4280 raft_consensus.cc:2454] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f13c6bc4d44b8f9e48c5afe9947fe5 in term 1.
I20260929 16:40:00.118340 4081 leader_election.cc:302] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f13c6bc4d44b8f9e48c5afe9947fe5, e81d6a385fe34e28ae4e45bfb99ab41d; no voters:
I20260929 16:40:00.118623 4484 raft_consensus.cc:2781] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:00.119046 4484 raft_consensus.cc:686] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 26f13c6bc4d44b8f9e48c5afe9947fe5, State: Running, Role: LEADER
I20260929 16:40:00.119534 4484 consensus_queue.cc:237] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } }
I20260929 16:40:00.124126 4001 catalog_manager.cc:5495] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f13c6bc4d44b8f9e48c5afe9947fe5 (127.17.205.1). New cstate: current_term: 1 leader_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f13c6bc4d44b8f9e48c5afe9947fe5" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 40843 } health_report { overall_health: HEALTHY } } }
I20260929 16:40:00.154625 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.148381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.17.205.1:40843)
0929 16:40:00.148574 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.148579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.148596 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:40:00.148698 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.148702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.148783 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.148903 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:00.149219 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.149927 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.151730 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.152358 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.152488 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.153496 (+ 1008us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:00.154136 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:00.154281 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.154386 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.154428 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:40:00.162654 4280 raft_consensus.cc:1261] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 1 FOLLOWER]: Refusing update from remote peer 26f13c6bc4d44b8f9e48c5afe9947fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:00.162679 4413 raft_consensus.cc:1261] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 1 FOLLOWER]: Refusing update from remote peer 26f13c6bc4d44b8f9e48c5afe9947fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:00.163614 4487 consensus_queue.cc:1035] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81d6a385fe34e28ae4e45bfb99ab41d" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:00.163941 4484 consensus_queue.cc:1035] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8985825c32a468188399f8cf41f54cf" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:00.173628 4490 mvcc.cc:205] Tried to move back new op lower bound from 7334707200656998400 to 7334707200493043712. Current Snapshot: MvccSnapshot[applied={T|T < 7334707200656998400}]
I20260929 16:40:00.176939 4491 mvcc.cc:205] Tried to move back new op lower bound from 7334707200656998400 to 7334707200493043712. Current Snapshot: MvccSnapshot[applied={T|T < 7334707200656998400}]
I20260929 16:40:00.187775 4001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:00.188069 4001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:00.191807 4001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a2439122b8466e87aa3fa4b1a9ca72: Sending DeleteTablet for 3 replicas of tablet 90b175d1d02140418d899300d8fb546b
I20260929 16:40:00.193049 4393 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b175d1d02140418d899300d8fb546b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:00 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:38680
I20260929 16:40:00.193055 4260 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b175d1d02140418d899300d8fb546b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:00 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:49436
I20260929 16:40:00.194056 4126 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b175d1d02140418d899300d8fb546b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:00 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:56536
I20260929 16:40:00.195688 4500 tablet_replica.cc:330] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: stopping tablet replica
I20260929 16:40:00.196039 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4063
I20260929 16:40:00.196141 4500 raft_consensus.cc:2229] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:00.196652 4500 raft_consensus.cc:2258] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:00.197355 4501 tablet_replica.cc:330] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: stopping tablet replica
I20260929 16:40:00.197780 4501 raft_consensus.cc:2229] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:40:00.198494 4501 raft_consensus.cc:2258] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:00.199530 4499 tablet_replica.cc:330] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: stopping tablet replica
I20260929 16:40:00.199959 4499 raft_consensus.cc:2229] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:00.200484 4499 raft_consensus.cc:2258] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:00.201284 4500 ts_tablet_manager.cc:1905] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:00.202939 4499 ts_tablet_manager.cc:1905] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:00.208258 4500 ts_tablet_manager.cc:1918] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:00.208422 4500 log.cc:1196] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/90b175d1d02140418d899300d8fb546b
Sep 29 16:40:00 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:40:00.208943 4500 ts_tablet_manager.cc:1939] T 90b175d1d02140418d899300d8fb546b P c8985825c32a468188399f8cf41f54cf: Deleting consensus metadata
I20260929 16:40:00.210291 4499 ts_tablet_manager.cc:1918] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:00.210466 4499 log.cc:1196] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/90b175d1d02140418d899300d8fb546b
I20260929 16:40:00.210983 4499 ts_tablet_manager.cc:1939] T 90b175d1d02140418d899300d8fb546b P e81d6a385fe34e28ae4e45bfb99ab41d: Deleting consensus metadata
I20260929 16:40:00.211925 3989 catalog_manager.cc:4841] TS c8985825c32a468188399f8cf41f54cf (127.17.205.3:32837): tablet 90b175d1d02140418d899300d8fb546b (table test-table [id=112b8e48808945208578cddfb5085d68]) successfully deleted
I20260929 16:40:00.212793 3988 catalog_manager.cc:4841] TS e81d6a385fe34e28ae4e45bfb99ab41d (127.17.205.2:46119): tablet 90b175d1d02140418d899300d8fb546b (table test-table [id=112b8e48808945208578cddfb5085d68]) successfully deleted
I20260929 16:40:00.212870 4501 ts_tablet_manager.cc:1905] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:00.218797 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.197222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.17.205.1:40843)
0929 16:40:00.197402 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.197409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.197806 (+ 397us) server_negotiation.cc:407] Connection header received
0929 16:40:00.198152 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.198158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.198245 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.198326 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:00.200363 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.201529 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.203747 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.204447 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.204551 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.210229 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:00.210246 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:00.210266 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:00.210324 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:00.215417 (+ 5093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.216361 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.216370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.216374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.216534 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.216971 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.216977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.216980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.217467 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:00.217821 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.218132 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.218405 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":63}
I20260929 16:40:00.219602 4186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:00.316286 4501 ts_tablet_manager.cc:1918] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:00.316568 4501 log.cc:1196] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/90b175d1d02140418d899300d8fb546b
I20260929 16:40:00.317075 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4063
I20260929 16:40:00.317166 4501 ts_tablet_manager.cc:1939] T 90b175d1d02140418d899300d8fb546b P 26f13c6bc4d44b8f9e48c5afe9947fe5: Deleting consensus metadata
W20260929 16:40:00.333432 3989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:40843 (error 108)
I20260929 16:40:00.333876 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4197
I20260929 16:40:00.335330 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.334801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56568)
0929 16:40:00.335004 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.335115 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:40:00.335734 3989 catalog_manager.cc:4584] TS 26f13c6bc4d44b8f9e48c5afe9947fe5 (127.17.205.1:40843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90b175d1d02140418d899300d8fb546b: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Sep 29 16:40:00 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:40:00.350005 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.334762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.17.205.2:46119)
0929 16:40:00.335053 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.335059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.335080 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:40:00.335270 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.335276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.335369 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.335486 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:00.336822 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.337606 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.338887 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.339369 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.339479 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.343998 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:00.344010 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:00.344026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:00.344071 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:00.347120 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.347907 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.347914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.347917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.348050 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.348437 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.348441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.348443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.348868 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:00.349160 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.349457 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.349680 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260929 16:40:00.350678 4320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:00.389253 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.388812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56576)
0929 16:40:00.389013 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.389110 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:40:00.390250 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.389887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56580)
0929 16:40:00.390039 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.390110 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:40:00.440061 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4197
I20260929 16:40:00.450896 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.450401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56592)
0929 16:40:00.450636 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.450738 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 16:40:00.452060 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.451667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56600)
0929 16:40:00.451857 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.451931 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:40:00.454519 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4330
Sep 29 16:40:00 dist-test-slave-3h01 krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:40:00.469630 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.455248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.17.205.3:32837)
0929 16:40:00.455428 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:00.455434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:00.455453 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:40:00.455598 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:00.455604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:00.455682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:00.455761 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:00.457050 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.457778 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.459147 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:00.459721 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:00.459834 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:00.464110 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:00.464127 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:00.464144 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:00.464179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:00.466718 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.467530 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.467541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.467544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.467697 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:00.468079 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:00.468084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:00.468086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:00.468454 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:00.468734 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:00.469064 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 16:40:00.469298 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:40:00.470359 4453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:00.532766 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.532304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56610)
0929 16:40:00.532520 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.532614 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:40:00.533967 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:00.533521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:40843 (local address 127.0.0.1:56622)
0929 16:40:00.533732 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:00.533829 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 16:40:00.560431 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4330
I20260929 16:40:00.574218 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 3970
I20260929 16:40:00.575268 4032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:00.684733 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 3970
2026-09-29T16:40:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4127 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:00Z Disabled control of system clock
I20260929 16:40:00.755854 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43713
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:43713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:40:01.080593 4511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:01.080891 4511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:01.080971 4511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:01.088521 4511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:01.088650 4511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:01.088690 4511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:01.088748 4511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:01.088792 4511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:01.098233 4511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:43713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:01.099565 4511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:01.101332 4511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:01.112705 4516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:01.112668 4517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:01.112736 4519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:01.113400 4511 server_base.cc:1034] running on GCE node
I20260929 16:40:01.114162 4511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:01.115620 4511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:01.116820 4511 hybrid_clock.cc:648] HybridClock initialized: now 1790700001116776 us; error 39 us; skew 500 ppm
I20260929 16:40:01.117141 4511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:01.119329 4511 webserver.cc:460] Webserver started at http://127.17.205.62:33607/ using document root <none> and password file <none>
I20260929 16:40:01.120004 4511 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:01.120102 4511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:01.120404 4511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:01.123337 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "1826d400ebf74ca282cc22284b4e8f3b"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "f1c2f6e36768fecca8ba62605db69fdb"
server_key_iv: "cae212249a63675c6555d1310c6c6ece"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.124135 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "1826d400ebf74ca282cc22284b4e8f3b"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "f1c2f6e36768fecca8ba62605db69fdb"
server_key_iv: "cae212249a63675c6555d1310c6c6ece"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.129164 4511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:40:01.132817 4525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:01.134222 4511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:01.134413 4511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "1826d400ebf74ca282cc22284b4e8f3b"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "f1c2f6e36768fecca8ba62605db69fdb"
server_key_iv: "cae212249a63675c6555d1310c6c6ece"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.134642 4511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:01.154898 4511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:01.156144 4511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:01.156522 4511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:01.174917 4511 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:43713
I20260929 16:40:01.174952 4576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:43713 every 8 connection(s)
I20260929 16:40:01.176494 4511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:01.178375 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4511
I20260929 16:40:01.178553 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:01.178990 18228 external_mini_cluster.cc:1428] Setting key dbe8dcc94d42d4e68290484a779cb5f1
I20260929 16:40:01.180722 4577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:01.193547 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:01.180967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.17.205.62:43713)
0929 16:40:01.181994 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:01.182005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:01.182050 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:40:01.183795 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:01.183819 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:01.184364 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:01.184761 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:01.185394 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.186770 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.188748 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.189760 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.189908 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:01.191087 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:01.191095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:01.191110 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:01.191779 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:01.191863 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:01.191890 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:01.191981 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:01.192220 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:40:01.192568 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":368,"threads_started":1}
I20260929 16:40:01.194407 4577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: Bootstrap starting.
I20260929 16:40:01.197654 4577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:01.198607 4577 log.cc:824] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:01.201386 4577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: No bootstrap required, opened a new log
I20260929 16:40:01.208421 4577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } }
I20260929 16:40:01.208792 4577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:01.208909 4577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1826d400ebf74ca282cc22284b4e8f3b, State: Initialized, Role: FOLLOWER
I20260929 16:40:01.209457 4577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } }
I20260929 16:40:01.209666 4577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:01.209794 4577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:01.209965 4577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:01.212107 4577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } }
I20260929 16:40:01.212581 4577 leader_election.cc:302] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1826d400ebf74ca282cc22284b4e8f3b; no voters:
I20260929 16:40:01.213146 4577 leader_election.cc:288] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:01.213310 4582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:01.214046 4582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [term 1 LEADER]: Becoming Leader. State: Replica: 1826d400ebf74ca282cc22284b4e8f3b, State: Running, Role: LEADER
I20260929 16:40:01.214563 4582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } }
I20260929 16:40:01.215065 4577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:01.217554 4584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1826d400ebf74ca282cc22284b4e8f3b. Latest consensus state: current_term: 1 leader_uuid: "1826d400ebf74ca282cc22284b4e8f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } } }
I20260929 16:40:01.217582 4583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1826d400ebf74ca282cc22284b4e8f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1826d400ebf74ca282cc22284b4e8f3b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43713 } } }
I20260929 16:40:01.218022 4583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:01.218029 4584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:01.220961 4588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:01.226626 4588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:01.235846 4588 catalog_manager.cc:1347] Generated new cluster ID: ef7920deea0c41c1bb2f930fc912f902
I20260929 16:40:01.235957 4588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:01.262192 4588 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:01.263773 4588 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:01.269825 4588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: Generated new TSK 0
I20260929 16:40:01.270623 4588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:01.280010 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--builtin_ntp_servers=127.17.205.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:40:01.607827 4601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:01.608120 4601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:01.608232 4601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:01.615589 4601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:01.615705 4601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:01.615844 4601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:40:01.625552 4601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:01.626940 4601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:01.628742 4601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:01.641796 4606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:01.641846 4607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:01.642020 4609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:01.642820 4601 server_base.cc:1034] running on GCE node
I20260929 16:40:01.643541 4601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:01.644620 4601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:01.645850 4601 hybrid_clock.cc:648] HybridClock initialized: now 1790700001645800 us; error 49 us; skew 500 ppm
I20260929 16:40:01.646185 4601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:01.648509 4601 webserver.cc:460] Webserver started at http://127.17.205.1:41129/ using document root <none> and password file <none>
I20260929 16:40:01.649225 4601 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:01.649326 4601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:01.649623 4601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:01.652653 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "5528c989e30bae7cb5663279a31e7614"
server_key_iv: "a400ba744f6408eb59293756c531944c"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.653417 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "5528c989e30bae7cb5663279a31e7614"
server_key_iv: "a400ba744f6408eb59293756c531944c"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.658583 4601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:40:01.662194 4615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:01.663759 4601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:01.663932 4601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb"
format_stamp: "Formatted at 2026-09-29 16:40:01 on dist-test-slave-3h01"
server_key: "5528c989e30bae7cb5663279a31e7614"
server_key_iv: "a400ba744f6408eb59293756c531944c"
server_key_version: "encryptionkey@0"
I20260929 16:40:01.664146 4601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:01.688951 4601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:01.689836 4601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:01.690126 4601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:01.691105 4601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:01.692648 4601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:01.692752 4601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:01.692874 4601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:01.692951 4601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:01.722903 4601 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:41971
I20260929 16:40:01.722935 4727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:41971 every 8 connection(s)
I20260929 16:40:01.724349 4601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:40:01.725914 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4601
I20260929 16:40:01.726092 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:40:01.726640 18228 external_mini_cluster.cc:1428] Setting key 7f02e3a3c92184569f4c185389345c3e
I20260929 16:40:01.729717 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--builtin_ntp_servers=127.17.205.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:40:01.737401 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:01.727198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:38029 (local address 127.17.205.62:43713)
0929 16:40:01.727762 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:01.727769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:01.729157 (+ 1388us) server_negotiation.cc:407] Connection header received
0929 16:40:01.731012 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:01.731018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:01.731099 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:01.731193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:01.732348 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.733045 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.734672 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.735078 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.735188 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:01.736643 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:01.736649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:01.736652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:01.736672 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:01.736725 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:01.736739 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:01.736820 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:01.737065 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:40:01.737151 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
I20260929 16:40:01.738811 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:01.727974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43713 (local address 127.17.205.1:38029)
0929 16:40:01.729013 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:01.729053 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:01.730742 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:01.731496 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:01.731505 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:01.731603 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:01.732184 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:01.732199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.733193 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.733197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:01.734528 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:01.734541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:01.735321 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:01.735328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:01.735553 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:01.736254 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:01.736257 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:01.736317 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:01.736429 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:01.736846 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:01.736857 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:01.736865 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:40:01.737047 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:40:01.737466 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":288,"threads_started":1}
I20260929 16:40:01.740366 4728 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43713
I20260929 16:40:01.740685 4728 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:01.741582 4728 heartbeater.cc:511] Master 127.17.205.62:43713 requested a full tablet report, sending...
I20260929 16:40:01.743752 4542 ts_manager.cc:194] Registered new tserver with Master: c8de1cf32a2e4c1fb4aad8f9921f39cb (127.17.205.1:41971)
I20260929 16:40:01.745293 4542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.1:38029
W20260929 16:40:02.050312 4732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:02.050606 4732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:02.050740 4732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:02.058146 4732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:02.058264 4732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:02.058399 4732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:40:02.067791 4732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:02.069144 4732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:02.071089 4732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:02.084445 4737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:02.084407 4740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:02.084467 4738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:02.085074 4732 server_base.cc:1034] running on GCE node
I20260929 16:40:02.085741 4732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:02.086725 4732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:02.087966 4732 hybrid_clock.cc:648] HybridClock initialized: now 1790700002087909 us; error 43 us; skew 500 ppm
I20260929 16:40:02.088325 4732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:02.090658 4732 webserver.cc:460] Webserver started at http://127.17.205.2:46005/ using document root <none> and password file <none>
I20260929 16:40:02.091436 4732 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:02.091552 4732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:02.091861 4732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:02.094920 4732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "dae1ab43660049bdbb5d6af2a5725512"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "defc0d88395dc5533803c90f6f973feb"
server_key_iv: "69c20c54429b3f5afa0a0a8381eb9aa2"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.095796 4732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "dae1ab43660049bdbb5d6af2a5725512"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "defc0d88395dc5533803c90f6f973feb"
server_key_iv: "69c20c54429b3f5afa0a0a8381eb9aa2"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.101310 4732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:02.105077 4746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.106451 4732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:40:02.106606 4732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "dae1ab43660049bdbb5d6af2a5725512"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "defc0d88395dc5533803c90f6f973feb"
server_key_iv: "69c20c54429b3f5afa0a0a8381eb9aa2"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.106810 4732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:02.126729 4732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:02.127667 4732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:02.127975 4732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:02.129000 4732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:02.130543 4732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:02.130637 4732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.130774 4732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:02.130846 4732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.162942 4732 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:38753
I20260929 16:40:02.162989 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:38753 every 8 connection(s)
I20260929 16:40:02.164335 4732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:40:02.174412 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4732
I20260929 16:40:02.174608 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:40:02.175055 18228 external_mini_cluster.cc:1428] Setting key f4d627a21377ef791229e32545bd15c1
I20260929 16:40:02.178162 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--builtin_ntp_servers=127.17.205.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:40:02.178236 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.166892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:52463 (local address 127.17.205.62:43713)
0929 16:40:02.167054 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.167059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.168637 (+ 1578us) server_negotiation.cc:407] Connection header received
0929 16:40:02.170553 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.170559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.170633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.170714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:02.171950 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.172680 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.174499 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.175104 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.175250 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.177356 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:02.177363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:02.177368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:02.177390 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:02.177444 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:02.177461 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:02.177548 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.177819 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.177924 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:40:02.179615 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.167528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43713 (local address 127.17.205.2:52463)
0929 16:40:02.168477 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:02.168516 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:02.170231 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:02.170935 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:02.170943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:02.171044 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:02.171768 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.171797 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.172812 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.172816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.174264 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.174282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.175418 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.175425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.175735 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.176982 (+ 1247us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:02.176984 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:02.177028 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:02.177130 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:02.177567 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:02.177579 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:02.177588 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:40:02.177776 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:40:02.178235 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":268,"threads_started":1}
I20260929 16:40:02.181102 4859 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43713
I20260929 16:40:02.181420 4859 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:02.182286 4859 heartbeater.cc:511] Master 127.17.205.62:43713 requested a full tablet report, sending...
I20260929 16:40:02.183890 4542 ts_manager.cc:194] Registered new tserver with Master: dae1ab43660049bdbb5d6af2a5725512 (127.17.205.2:38753)
I20260929 16:40:02.184978 4542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.2:52463
W20260929 16:40:02.502655 4862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:02.503010 4862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:02.503146 4862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:02.510519 4862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:02.510624 4862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:02.510764 4862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:40:02.520270 4862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:02.521605 4862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:02.523499 4862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:02.536808 4867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:02.537248 4870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:02.537300 4868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:02.537611 4862 server_base.cc:1034] running on GCE node
I20260929 16:40:02.538249 4862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:02.539393 4862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:02.540607 4862 hybrid_clock.cc:648] HybridClock initialized: now 1790700002540553 us; error 46 us; skew 500 ppm
I20260929 16:40:02.540977 4862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:02.543435 4862 webserver.cc:460] Webserver started at http://127.17.205.3:41987/ using document root <none> and password file <none>
I20260929 16:40:02.544173 4862 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:02.544275 4862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:02.544592 4862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:02.547703 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "66b95fd9095e49fa9c4048e7066bb6dd"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "0746373580f4037352b52623bbdd38ea"
server_key_iv: "30e0c48b5be77d7d18addf2d52a0bfd0"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.548574 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "66b95fd9095e49fa9c4048e7066bb6dd"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "0746373580f4037352b52623bbdd38ea"
server_key_iv: "30e0c48b5be77d7d18addf2d52a0bfd0"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.554050 4862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:40:02.557574 4876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.558911 4862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:40:02.559055 4862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "66b95fd9095e49fa9c4048e7066bb6dd"
format_stamp: "Formatted at 2026-09-29 16:40:02 on dist-test-slave-3h01"
server_key: "0746373580f4037352b52623bbdd38ea"
server_key_iv: "30e0c48b5be77d7d18addf2d52a0bfd0"
server_key_version: "encryptionkey@0"
I20260929 16:40:02.559295 4862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:02.577989 4862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:02.578830 4862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:02.579093 4862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:02.580081 4862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:02.581504 4862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:02.581589 4862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.581678 4862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:02.581760 4862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:02.611634 4862 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:44225
I20260929 16:40:02.611666 4988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:44225 every 8 connection(s)
I20260929 16:40:02.612989 4862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:40:02.615106 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 4862
I20260929 16:40:02.615332 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:40:02.615877 18228 external_mini_cluster.cc:1428] Setting key 2d6c1d1faade2959789f0c0991f712c0
I20260929 16:40:02.625614 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.615881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:56703 (local address 127.17.205.62:43713)
0929 16:40:02.616052 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.616056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.617780 (+ 1724us) server_negotiation.cc:407] Connection header received
0929 16:40:02.619626 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.619630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.619703 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.619775 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:02.620829 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.621602 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.623200 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.623627 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.623742 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.624899 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:02.624904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:02.624907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:02.624924 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:02.624955 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:02.624966 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:02.625034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.625302 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.625379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:40:02.626722 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.616562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:43713 (local address 127.17.205.3:56703)
0929 16:40:02.617624 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:02.617670 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:02.619366 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:02.619986 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:02.619995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:02.620097 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:02.620702 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.620718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.621738 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.621741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.623084 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.623096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.623812 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.623816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.624012 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.624616 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:02.624618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:02.624656 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:02.624739 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:02.625088 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:02.625097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:02.625105 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:40:02.625291 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:40:02.625605 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":324,"threads_started":1}
I20260929 16:40:02.628170 4989 heartbeater.cc:347] Connected to a master server at 127.17.205.62:43713
I20260929 16:40:02.628499 4989 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:02.629287 4989 heartbeater.cc:511] Master 127.17.205.62:43713 requested a full tablet report, sending...
I20260929 16:40:02.630914 4541 ts_manager.cc:194] Registered new tserver with Master: 66b95fd9095e49fa9c4048e7066bb6dd (127.17.205.3:44225)
I20260929 16:40:02.631773 4541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.3:56703
I20260929 16:40:02.642990 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:40:02.652617 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.646920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.17.205.62:43713)
0929 16:40:02.647110 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.647116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.647477 (+ 361us) server_negotiation.cc:407] Connection header received
0929 16:40:02.647648 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.647652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.647733 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.647814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:02.648277 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.648975 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.650286 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.650703 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.650821 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.651935 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:02.651940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:02.651943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:02.651959 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:02.651990 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:02.652001 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:02.652135 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.652312 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.652368 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:40:02.656299 4541 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:40:02.663101 4541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:40:02.692006 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.679696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.17.205.1:41971)
0929 16:40:02.680278 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.680285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.681521 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 16:40:02.682027 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.682040 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.682417 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.682680 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:02.683706 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.685332 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.688142 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.689794 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.689918 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.690217 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.691368 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.691584 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":1409,"threads_started":1}
I20260929 16:40:02.692656 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.679189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54612)
0929 16:40:02.681398 (+ 2209us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:02.681419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:02.681632 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:02.683037 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:02.683043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:02.683071 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:02.683503 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.683515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.685507 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.685513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.687909 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.687926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.690443 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.690450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.690718 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.690778 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:40:02.690930 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:40:02.691154 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"thread_start_us":11601,"threads_started":1}
I20260929 16:40:02.696221 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.678234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38753 (local address 127.0.0.1:41004)
0929 16:40:02.679789 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:02.679829 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:02.680021 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:02.681054 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:02.681061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:02.681097 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:02.681549 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.681565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.683570 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.683576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.686145 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.686165 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.695427 (+ 9262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.695434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.695648 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.695682 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:40:02.695769 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:40:02.695916 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":1806,"threads_started":1}
I20260929 16:40:02.697105 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.678373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.17.205.2:38753)
0929 16:40:02.678947 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.678955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.679952 (+ 997us) server_negotiation.cc:407] Connection header received
0929 16:40:02.680348 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.680358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.680630 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.680851 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:02.681757 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.683384 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.686410 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.694844 (+ 8434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.694980 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.695290 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.696604 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.696790 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
I20260929 16:40:02.697036 4663 tablet_service.cc:1467] Processing CreateTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c (DEFAULT_TABLE table=test-table [id=c958c9d8e818450d8135d337c5a6f895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:02.700663 4794 tablet_service.cc:1467] Processing CreateTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c (DEFAULT_TABLE table=test-table [id=c958c9d8e818450d8135d337c5a6f895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:02.702011 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e498b39cc15d4963ae4fc56f6aa0c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:02.702186 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.680807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:44225 (local address 127.0.0.1:58344)
0929 16:40:02.690559 (+ 9752us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:02.690581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:02.690722 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:02.693772 (+ 3050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:02.693777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:02.693806 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:02.694069 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.694080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.696527 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.696532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.699123 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:02.699138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.701453 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.701460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:02.701627 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.701661 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:40:02.701747 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:40:02.701880 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9600,"thread_start_us":835,"threads_started":1}
I20260929 16:40:02.702317 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:02.681410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.17.205.3:44225)
0929 16:40:02.692779 (+ 11369us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:02.692796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:02.692834 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:40:02.692943 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:02.692951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:02.693243 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:02.693465 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:02.694747 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.696383 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.699825 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:02.701225 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:02.701354 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:02.701630 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:02.701820 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 16:40:02.701987 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11159,"thread_start_us":10991,"threads_started":1}
I20260929 16:40:02.705899 4924 tablet_service.cc:1467] Processing CreateTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c (DEFAULT_TABLE table=test-table [id=c958c9d8e818450d8135d337c5a6f895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:02.707108 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e498b39cc15d4963ae4fc56f6aa0c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:02.707871 4663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e498b39cc15d4963ae4fc56f6aa0c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:02.718282 5007 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Bootstrap starting.
I20260929 16:40:02.721652 5007 tablet_bootstrap.cc:654] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:02.722065 5008 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Bootstrap starting.
I20260929 16:40:02.725597 5008 tablet_bootstrap.cc:654] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:02.726817 5008 log.cc:824] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:02.728976 5007 log.cc:824] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:02.734412 5009 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Bootstrap starting.
I20260929 16:40:02.736713 5007 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: No bootstrap required, opened a new log
I20260929 16:40:02.737090 5007 ts_tablet_manager.cc:1397] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260929 16:40:02.739523 5009 tablet_bootstrap.cc:654] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:02.739809 5008 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: No bootstrap required, opened a new log
I20260929 16:40:02.740115 5008 ts_tablet_manager.cc:1397] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260929 16:40:02.740826 5009 log.cc:824] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:02.744107 5009 tablet_bootstrap.cc:492] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: No bootstrap required, opened a new log
I20260929 16:40:02.744410 5009 ts_tablet_manager.cc:1397] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:40:02.745172 5007 raft_consensus.cc:348] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.745558 5007 raft_consensus.cc:374] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:02.745652 5007 raft_consensus.cc:729] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae1ab43660049bdbb5d6af2a5725512, State: Initialized, Role: FOLLOWER
I20260929 16:40:02.746238 5007 consensus_queue.cc:260] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.747373 4859 heartbeater.cc:503] Master 127.17.205.62:43713 was elected leader, sending a full tablet report...
I20260929 16:40:02.748008 5007 ts_tablet_manager.cc:1428] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 16:40:02.749090 4728 heartbeater.cc:503] Master 127.17.205.62:43713 was elected leader, sending a full tablet report...
I20260929 16:40:02.750568 5008 raft_consensus.cc:348] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.751000 5008 raft_consensus.cc:374] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:02.751098 5008 raft_consensus.cc:729] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b95fd9095e49fa9c4048e7066bb6dd, State: Initialized, Role: FOLLOWER
I20260929 16:40:02.751847 5008 consensus_queue.cc:260] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.753587 5008 ts_tablet_manager.cc:1428] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 16:40:02.753664 4989 heartbeater.cc:503] Master 127.17.205.62:43713 was elected leader, sending a full tablet report...
I20260929 16:40:02.755710 5009 raft_consensus.cc:348] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.756106 5009 raft_consensus.cc:374] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:02.756193 5009 raft_consensus.cc:729] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8de1cf32a2e4c1fb4aad8f9921f39cb, State: Initialized, Role: FOLLOWER
I20260929 16:40:02.756762 5009 consensus_queue.cc:260] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:02.758260 5009 ts_tablet_manager.cc:1428] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
W20260929 16:40:02.864596 4990 tablet.cc:2369] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:02.916285 4860 tablet.cc:2369] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:02.978457 4729 tablet.cc:2369] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:03.005713 5014 raft_consensus.cc:482] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:40:03.005999 5014 raft_consensus.cc:504] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:03.007668 5014 leader_election.cc:288] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dae1ab43660049bdbb5d6af2a5725512 (127.17.205.2:38753), c8de1cf32a2e4c1fb4aad8f9921f39cb (127.17.205.1:41971)
I20260929 16:40:03.014366 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.008068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:41701 (local address 127.17.205.2:38753)
0929 16:40:03.008259 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.008265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.008299 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:40:03.008537 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.008543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.008630 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.008731 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:03.009212 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.010105 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.012535 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.013625 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.013786 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.013857 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.014030 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.014158 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:40:03.014454 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.007906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:38753 (local address 127.17.205.3:41701)
0929 16:40:03.008134 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.008157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:03.008285 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:03.008780 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:03.008804 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:03.008839 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:03.009068 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:03.009079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.010249 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.010253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:03.012389 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:03.012405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.013771 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.013775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:03.013895 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.013920 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:40:03.013986 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:40:03.014113 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":44,"spinlock_wait_cycles":51840}
I20260929 16:40:03.014816 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.008353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:44641 (local address 127.17.205.1:41971)
0929 16:40:03.008545 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.008551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.008890 (+ 339us) server_negotiation.cc:407] Connection header received
0929 16:40:03.009063 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.009066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.009135 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.009224 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:03.009663 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.010547 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.012828 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.013977 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.014124 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.014183 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.014460 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.014585 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:40:03.015159 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.008051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.17.205.3:44641)
0929 16:40:03.008792 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.008808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:03.008950 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:03.009241 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:03.009244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:03.009263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:03.009524 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:03.009532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.010686 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.010692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:03.012670 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:03.012680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.014123 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.014128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:03.014304 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.014337 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:40:03.014422 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:40:03.014836 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":34,"spinlock_wait_cycles":768,"thread_start_us":413,"threads_started":1}
I20260929 16:40:03.015528 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e498b39cc15d4963ae4fc56f6aa0c55c" candidate_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae1ab43660049bdbb5d6af2a5725512" is_pre_election: true
I20260929 16:40:03.016105 4814 raft_consensus.cc:2454] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b95fd9095e49fa9c4048e7066bb6dd in term 0.
I20260929 16:40:03.016345 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e498b39cc15d4963ae4fc56f6aa0c55c" candidate_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" is_pre_election: true
I20260929 16:40:03.016801 4878 leader_election.cc:302] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b95fd9095e49fa9c4048e7066bb6dd, dae1ab43660049bdbb5d6af2a5725512; no voters:
I20260929 16:40:03.016880 4683 raft_consensus.cc:2454] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b95fd9095e49fa9c4048e7066bb6dd in term 0.
I20260929 16:40:03.017204 5014 raft_consensus.cc:2781] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:40:03.017387 5014 raft_consensus.cc:482] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:40:03.017496 5014 raft_consensus.cc:3037] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:03.019701 5014 raft_consensus.cc:504] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:03.020244 5014 leader_election.cc:288] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [CANDIDATE]: Term 1 election: Requested vote from peers dae1ab43660049bdbb5d6af2a5725512 (127.17.205.2:38753), c8de1cf32a2e4c1fb4aad8f9921f39cb (127.17.205.1:41971)
I20260929 16:40:03.020668 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e498b39cc15d4963ae4fc56f6aa0c55c" candidate_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb"
I20260929 16:40:03.020684 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e498b39cc15d4963ae4fc56f6aa0c55c" candidate_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae1ab43660049bdbb5d6af2a5725512"
I20260929 16:40:03.020871 4683 raft_consensus.cc:3037] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:03.020887 4814 raft_consensus.cc:3037] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:03.023962 4683 raft_consensus.cc:2454] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b95fd9095e49fa9c4048e7066bb6dd in term 1.
I20260929 16:40:03.023975 4814 raft_consensus.cc:2454] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b95fd9095e49fa9c4048e7066bb6dd in term 1.
I20260929 16:40:03.024359 4878 leader_election.cc:302] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b95fd9095e49fa9c4048e7066bb6dd, dae1ab43660049bdbb5d6af2a5725512; no voters:
I20260929 16:40:03.024667 5014 raft_consensus.cc:2781] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:03.025110 5014 raft_consensus.cc:686] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 1 LEADER]: Becoming Leader. State: Replica: 66b95fd9095e49fa9c4048e7066bb6dd, State: Running, Role: LEADER
I20260929 16:40:03.025573 5014 consensus_queue.cc:237] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } }
I20260929 16:40:03.030246 4542 catalog_manager.cc:5495] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 66b95fd9095e49fa9c4048e7066bb6dd (127.17.205.3). New cstate: current_term: 1 leader_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b95fd9095e49fa9c4048e7066bb6dd" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 44225 } health_report { overall_health: HEALTHY } } }
I20260929 16:40:03.077382 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.071679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.17.205.3:44225)
0929 16:40:03.071937 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.071944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.071969 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:40:03.072100 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.072107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.072225 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.072403 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:03.072849 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.073628 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.075017 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.075514 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.075626 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.076185 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:03.076734 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:03.076889 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.077043 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.077092 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:40:03.085613 4683 raft_consensus.cc:1261] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb [term 1 FOLLOWER]: Refusing update from remote peer 66b95fd9095e49fa9c4048e7066bb6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:03.085611 4814 raft_consensus.cc:1261] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 1 FOLLOWER]: Refusing update from remote peer 66b95fd9095e49fa9c4048e7066bb6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:03.086549 5017 consensus_queue.cc:1035] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8de1cf32a2e4c1fb4aad8f9921f39cb" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:03.086885 5014 consensus_queue.cc:1035] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dae1ab43660049bdbb5d6af2a5725512" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:03.113261 4541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:03.113547 4541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:03.117354 4541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1826d400ebf74ca282cc22284b4e8f3b: Sending DeleteTablet for 3 replicas of tablet e498b39cc15d4963ae4fc56f6aa0c55c
I20260929 16:40:03.118894 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:03 UTC) from {username='slave'} at 127.0.0.1:58344
I20260929 16:40:03.118888 4663 tablet_service.cc:1514] Processing DeleteTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:03 UTC) from {username='slave'} at 127.0.0.1:54612
I20260929 16:40:03.118934 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet e498b39cc15d4963ae4fc56f6aa0c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:03 UTC) from {username='slave'} at 127.0.0.1:41004
I20260929 16:40:03.120709 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4601
I20260929 16:40:03.130693 5031 tablet_replica.cc:330] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: stopping tablet replica
I20260929 16:40:03.131230 5031 raft_consensus.cc:2229] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:03.131810 5031 raft_consensus.cc:2258] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:03.134702 5031 ts_tablet_manager.cc:1905] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:03.140305 5029 tablet_replica.cc:330] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: stopping tablet replica
I20260929 16:40:03.140856 5029 raft_consensus.cc:2229] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:40:03.141672 5029 raft_consensus.cc:2258] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:03.144765 5031 ts_tablet_manager.cc:1918] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:03.144951 5031 log.cc:1196] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/e498b39cc15d4963ae4fc56f6aa0c55c
I20260929 16:40:03.145439 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.122360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.17.205.1:41971)
0929 16:40:03.122580 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.122587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.132983 (+ 10396us) server_negotiation.cc:407] Connection header received
0929 16:40:03.133174 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.133180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.133331 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.133449 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:03.136227 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.137709 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.140615 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.141394 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.141535 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.143182 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:03.143194 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:03.143248 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:03.144212 (+ 964us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:03.144299 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:03.144331 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:03.144445 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.144672 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.144827 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:40:03.145617 5031 ts_tablet_manager.cc:1939] T e498b39cc15d4963ae4fc56f6aa0c55c P dae1ab43660049bdbb5d6af2a5725512: Deleting consensus metadata
I20260929 16:40:03.147022 4723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:03.147676 4527 catalog_manager.cc:4841] TS dae1ab43660049bdbb5d6af2a5725512 (127.17.205.2:38753): tablet e498b39cc15d4963ae4fc56f6aa0c55c (table test-table [id=c958c9d8e818450d8135d337c5a6f895]) successfully deleted
I20260929 16:40:03.149758 5029 ts_tablet_manager.cc:1905] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:03.158183 5029 ts_tablet_manager.cc:1918] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:03.158396 5029 log.cc:1196] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/e498b39cc15d4963ae4fc56f6aa0c55c
I20260929 16:40:03.158985 5029 ts_tablet_manager.cc:1939] T e498b39cc15d4963ae4fc56f6aa0c55c P 66b95fd9095e49fa9c4048e7066bb6dd: Deleting consensus metadata
I20260929 16:40:03.161015 4527 catalog_manager.cc:4841] TS 66b95fd9095e49fa9c4048e7066bb6dd (127.17.205.3:44225): tablet e498b39cc15d4963ae4fc56f6aa0c55c (table test-table [id=c958c9d8e818450d8135d337c5a6f895]) successfully deleted
I20260929 16:40:03.257704 5030 tablet_replica.cc:330] T e498b39cc15d4963ae4fc56f6aa0c55c P c8de1cf32a2e4c1fb4aad8f9921f39cb: stopping tablet replica
I20260929 16:40:03.257664 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4601
W20260929 16:40:03.272931 4528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.205.1:41971 (error 108)
I20260929 16:40:03.273161 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4732
I20260929 16:40:03.275162 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.274322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54632)
0929 16:40:03.274809 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.274945 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":200,"threads_started":1}
W20260929 16:40:03.275550 4528 catalog_manager.cc:4584] TS c8de1cf32a2e4c1fb4aad8f9921f39cb (127.17.205.1:41971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e498b39cc15d4963ae4fc56f6aa0c55c: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
I20260929 16:40:03.281152 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.274031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.17.205.2:38753)
0929 16:40:03.274231 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.274237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.274260 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:40:03.274425 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.274430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.274527 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.274632 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:03.275173 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.276316 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.277649 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.278302 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.278439 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.279516 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:03.279528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:03.279548 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:03.280358 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:03.280425 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:03.280450 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:03.280564 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.280644 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.280766 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:40:03.281889 4854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:03.329119 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.328562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54644)
0929 16:40:03.328814 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.328936 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:40:03.330384 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.329953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54658)
0929 16:40:03.330118 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.330207 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:40:03.370394 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4732
I20260929 16:40:03.384588 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4862
I20260929 16:40:03.391151 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.390629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54670)
0929 16:40:03.390869 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.390971 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:40:03.391736 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.385337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.17.205.3:44225)
0929 16:40:03.385524 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:03.385530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:03.385630 (+ 100us) server_negotiation.cc:407] Connection header received
0929 16:40:03.385813 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:03.385818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:03.385901 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:03.385985 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:03.386401 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.387237 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.388495 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:03.388908 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:03.389020 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:03.390130 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:03.390142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:03.390157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:03.390793 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:03.390859 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:03.390891 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:03.391000 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:03.391176 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:40:03.391441 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:40:03.392431 4984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:03.392606 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.392195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54672)
0929 16:40:03.392358 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.392448 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 16:40:03.473362 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.472878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54678)
0929 16:40:03.473096 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.473193 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:40:03.474570 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:03.474111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:41971 (local address 127.0.0.1:54688)
0929 16:40:03.474354 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:03.474429 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:40:03.510018 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4862
I20260929 16:40:03.526342 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 4511
I20260929 16:40:03.527523 4572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:03.636619 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 4511
2026-09-29T16:40:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2952 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:03 dist-test-slave-3h01 krb5kdc[5041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:03 dist-test-slave-3h01 krb5kdc[5041](info): set up 2 sockets
Sep 29 16:40:03 dist-test-slave-3h01 krb5kdc[5041](info): commencing operation
krb5kdc: starting...
W20260929 16:40:04.651595 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:04 dist-test-slave-3h01 krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:04.894508 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33643
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:33643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:05.223380 5054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:05.223706 5054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:05.223850 5054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:05.231124 5054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:05.231284 5054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:05.231364 5054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:05.231442 5054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:05.231500 5054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:05.241626 5054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:33643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:33643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:05.243057 5054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:05.244446 5054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:05.255960 5062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:05.256042 5059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:05.256068 5060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:05.257073 5054 server_base.cc:1034] running on GCE node
I20260929 16:40:05.257881 5054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:05.259402 5054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:05.260607 5054 hybrid_clock.cc:648] HybridClock initialized: now 1790700005260557 us; error 41 us; skew 500 ppm
I20260929 16:40:05.260951 5054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:05 dist-test-slave-3h01 krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700005, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:05.268707 5054 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:05.270344 5054 webserver.cc:460] Webserver started at http://127.17.205.62:40883/ using document root <none> and password file <none>
I20260929 16:40:05.271070 5054 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:05.271191 5054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:05.271535 5054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:05.274639 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "41af2aac424a4c83909618ece4545c9f"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "9d578e88ade49e23af1d3cb3d0033eb4"
server_key_iv: "29a98606463d58285e222e6aa4fcfc51"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.275463 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "41af2aac424a4c83909618ece4545c9f"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "9d578e88ade49e23af1d3cb3d0033eb4"
server_key_iv: "29a98606463d58285e222e6aa4fcfc51"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.280664 5054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:40:05.284199 5069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:05.285573 5054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:40:05.285732 5054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "41af2aac424a4c83909618ece4545c9f"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "9d578e88ade49e23af1d3cb3d0033eb4"
server_key_iv: "29a98606463d58285e222e6aa4fcfc51"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.285930 5054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:05.309566 5054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:05.313992 5054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:05.314270 5054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:05.331471 5054 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:33643
I20260929 16:40:05.331475 5120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:33643 every 8 connection(s)
I20260929 16:40:05.332846 5054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:05.336608 5121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:05.337965 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5054
I20260929 16:40:05.338169 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:05.338693 18228 external_mini_cluster.cc:1428] Setting key b77da4a287ceb40985371699fa29149e
I20260929 16:40:05.350155 5121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: Bootstrap starting.
Sep 29 16:40:05 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:05.353366 5121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:05.354411 5121 log.cc:824] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:05.357453 5121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: No bootstrap required, opened a new log
I20260929 16:40:05.361420 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:05.340974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.17.205.62:33643)
0929 16:40:05.341864 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:05.341885 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:05.341947 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:40:05.344088 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:05.344114 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:05.344580 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:05.344968 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:05.346473 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.347886 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.349204 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.349923 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.350100 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:05.354000 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:05.354008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:05.354024 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:05.354067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:05.357107 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:05.357885 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:05.357893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:05.357900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:05.358084 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:05.358490 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:05.358494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:05.358496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:05.359088 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:05.359421 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:05.359892 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 16:40:05.360357 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":372,"threads_started":1}
I20260929 16:40:05.365563 5121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } }
I20260929 16:40:05.365973 5121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:05.366063 5121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41af2aac424a4c83909618ece4545c9f, State: Initialized, Role: FOLLOWER
I20260929 16:40:05.366622 5121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } }
I20260929 16:40:05.366850 5121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:05.366973 5121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:05.367120 5121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:05.369582 5121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } }
I20260929 16:40:05.370097 5121 leader_election.cc:302] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41af2aac424a4c83909618ece4545c9f; no voters:
I20260929 16:40:05.370667 5121 leader_election.cc:288] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:05.370801 5126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:05.371418 5126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [term 1 LEADER]: Becoming Leader. State: Replica: 41af2aac424a4c83909618ece4545c9f, State: Running, Role: LEADER
I20260929 16:40:05.371939 5126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } }
I20260929 16:40:05.372717 5121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:05.374634 5128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41af2aac424a4c83909618ece4545c9f. Latest consensus state: current_term: 1 leader_uuid: "41af2aac424a4c83909618ece4545c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } } }
I20260929 16:40:05.375048 5128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:05.375741 5127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41af2aac424a4c83909618ece4545c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41af2aac424a4c83909618ece4545c9f" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 33643 } } }
I20260929 16:40:05.376109 5127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:05.380429 5133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:05.386929 5133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:05.395871 5133 catalog_manager.cc:1347] Generated new cluster ID: 46c170cc96fa42b98f517c0d3c0003aa
I20260929 16:40:05.395998 5133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:05.408413 5133 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:05.410035 5133 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:05.420971 5133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: Generated new TSK 0
I20260929 16:40:05.422081 5133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.205.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
Entry for principal kudu/127.17.205.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab.
I20260929 16:40:05.493700 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:05.853150 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:05.853447 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:05.853588 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:05.860917 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:05.861043 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:05.861192 5147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:40:05.870864 5147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.1.keytab
--principal=kudu/127.17.205.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:05.872174 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:05.873525 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:05.886082 5152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:05.886061 5155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:05.886132 5153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:05.886864 5147 server_base.cc:1034] running on GCE node
I20260929 16:40:05.887465 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:05.888348 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:05.889546 5147 hybrid_clock.cc:648] HybridClock initialized: now 1790700005889490 us; error 47 us; skew 500 ppm
I20260929 16:40:05.889871 5147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:05 dist-test-slave-3h01 krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700005, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:05.892948 5147 init.cc:377] Logged in from keytab as kudu/127.17.205.1@KRBTEST.COM (short username kudu)
I20260929 16:40:05.894308 5147 webserver.cc:460] Webserver started at http://127.17.205.1:46307/ using document root <none> and password file <none>
I20260929 16:40:05.894958 5147 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:05.895071 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:05.895398 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:05.898325 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "65d3c30355df4f0d90865939f531b8f0"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "f256ae4c97c9ba593e50a31c837f3f99"
server_key_iv: "dc5215a836132d1995076ec87eb0877e"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.899056 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "65d3c30355df4f0d90865939f531b8f0"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "f256ae4c97c9ba593e50a31c837f3f99"
server_key_iv: "dc5215a836132d1995076ec87eb0877e"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.904139 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 16:40:05.907503 5162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:05.908746 5147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:40:05.908902 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "65d3c30355df4f0d90865939f531b8f0"
format_stamp: "Formatted at 2026-09-29 16:40:05 on dist-test-slave-3h01"
server_key: "f256ae4c97c9ba593e50a31c837f3f99"
server_key_iv: "dc5215a836132d1995076ec87eb0877e"
server_key_version: "encryptionkey@0"
I20260929 16:40:05.909075 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:05.933070 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:05.937352 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:05.937629 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:05.938598 5147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:05.939994 5147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:05.940071 5147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:05.940192 5147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:05.940243 5147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:05.968963 5147 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:34881
I20260929 16:40:05.968995 5274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:34881 every 8 connection(s)
I20260929 16:40:05.970254 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:40:05.971583 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5147
I20260929 16:40:05.971760 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:40:05.972273 18228 external_mini_cluster.cc:1428] Setting key d87c8466bde39073147a8936a95515b3
Sep 29 16:40:05 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700005, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.1@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:05.992302 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:05.972773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:45605 (local address 127.17.205.62:33643)
0929 16:40:05.973310 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:05.973315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:05.974380 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 16:40:05.977025 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:05.977029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:05.977100 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:05.977195 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:05.979154 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.979941 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.982054 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.982481 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.982580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:05.986756 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:05.986762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:05.986765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:05.986801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:05.989187 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:05.990008 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:05.990012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:05.990014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:05.990133 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:05.990599 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:05.990604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:05.990606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:05.990890 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:05.991002 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:05.991871 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 16:40:05.992044 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":282,"threads_started":1}
I20260929 16:40:05.993705 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:05.973374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33643 (local address 127.17.205.1:45605)
0929 16:40:05.974228 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:05.974270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:05.976694 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:05.977476 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:05.977489 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:05.978189 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:05.978972 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:05.978997 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.980094 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.980099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:05.981920 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:05.981934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:05.982706 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:05.982713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:05.983000 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:05.983877 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:05.983915 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:05.986519 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:05.989341 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:05.989350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:05.989368 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:05.989911 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:05.990223 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:05.990228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:05.990232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:05.990493 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:05.991005 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:05.991013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:05.991486 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 16:40:05.991797 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 16:40:05.992348 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":314,"threads_started":1}
I20260929 16:40:05.995265 5275 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33643
I20260929 16:40:05.995571 5275 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:05.996330 5275 heartbeater.cc:511] Master 127.17.205.62:33643 requested a full tablet report, sending...
I20260929 16:40:05.998314 5086 ts_manager.cc:194] Registered new tserver with Master: 65d3c30355df4f0d90865939f531b8f0 (127.17.205.1:34881)
WARNING: no policy specified for kudu/127.17.205.2@KRBTEST.COM; defaulting to no policy
I20260929 16:40:05.999854 5086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.1@KRBTEST.COM'} at 127.17.205.1:45605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
Entry for principal kudu/127.17.205.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab.
I20260929 16:40:06.028051 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:06.352437 5281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:06.352746 5281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:06.352895 5281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:06.360391 5281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:06.360529 5281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:06.360688 5281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:40:06.370843 5281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.2.keytab
--principal=kudu/127.17.205.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:06.372278 5281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:06.373766 5281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:06.386749 5287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:06.386715 5289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:06.386783 5286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:06.387475 5281 server_base.cc:1034] running on GCE node
I20260929 16:40:06.388077 5281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:06.389061 5281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:06.390278 5281 hybrid_clock.cc:648] HybridClock initialized: now 1790700006390225 us; error 43 us; skew 500 ppm
I20260929 16:40:06.390601 5281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:06 dist-test-slave-3h01 krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700006, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:06.393784 5281 init.cc:377] Logged in from keytab as kudu/127.17.205.2@KRBTEST.COM (short username kudu)
I20260929 16:40:06.395450 5281 webserver.cc:460] Webserver started at http://127.17.205.2:40461/ using document root <none> and password file <none>
I20260929 16:40:06.396183 5281 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:06.396296 5281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:06.396565 5281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:06.399643 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "43afbe250d7241d8b5ff6090aae710e2"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "d07b4d9966af749c4ec59b992453efb5"
server_key_iv: "90326720046a4e385e3663fa1021de9b"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.400440 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "43afbe250d7241d8b5ff6090aae710e2"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "d07b4d9966af749c4ec59b992453efb5"
server_key_iv: "90326720046a4e385e3663fa1021de9b"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.405750 5281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 16:40:06.409314 5296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.410718 5281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:40:06.410880 5281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "43afbe250d7241d8b5ff6090aae710e2"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "d07b4d9966af749c4ec59b992453efb5"
server_key_iv: "90326720046a4e385e3663fa1021de9b"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.411098 5281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:06.433764 5281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:06.438232 5281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:06.438481 5281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:06.439467 5281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:06.440853 5281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:06.440932 5281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.441025 5281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:06.441066 5281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.471567 5281 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:36425
I20260929 16:40:06.471611 5408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:36425 every 8 connection(s)
I20260929 16:40:06.472977 5281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:40:06.481645 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5281
I20260929 16:40:06.481817 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:40:06.482246 18228 external_mini_cluster.cc:1428] Setting key fa5167b34c855eb664efb1b30e79c59f
Sep 29 16:40:06 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700006, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.2@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:06.493425 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:06.475592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:56925 (local address 127.17.205.62:33643)
0929 16:40:06.475784 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:06.475791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:06.477005 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 16:40:06.478815 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:06.478820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:06.478896 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:06.478983 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:06.480512 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.481262 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.483076 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.483624 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.483731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:06.487402 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:06.487408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:06.487412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:06.487453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:06.490431 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:06.491184 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:06.491189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:06.491192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:06.491375 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:06.491758 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:06.491763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:06.491766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:06.492089 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:06.492228 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:06.492965 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 16:40:06.493187 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:40:06.494827 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:06.476120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33643 (local address 127.17.205.2:56925)
0929 16:40:06.476835 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:06.476879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:06.478529 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:06.479185 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:06.479193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:06.479738 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:06.480345 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:06.480369 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.481424 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.481428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:06.482936 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:06.482951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.483849 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.483857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:06.484150 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:06.484884 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:06.484907 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:06.487130 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:06.490568 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:06.490575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:06.490589 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:06.491073 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:06.491466 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:06.491469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:06.491472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:06.491657 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:06.492233 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:06.492240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:06.492603 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 16:40:06.492849 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 16:40:06.493402 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
I20260929 16:40:06.496301 5409 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33643
I20260929 16:40:06.496627 5409 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:06.497485 5409 heartbeater.cc:511] Master 127.17.205.62:33643 requested a full tablet report, sending...
I20260929 16:40:06.499131 5086 ts_manager.cc:194] Registered new tserver with Master: 43afbe250d7241d8b5ff6090aae710e2 (127.17.205.2:36425)
I20260929 16:40:06.500013 5086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.2@KRBTEST.COM'} at 127.17.205.2:56925
WARNING: no policy specified for kudu/127.17.205.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
Entry for principal kudu/127.17.205.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab.
I20260929 16:40:06.536533 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:06.856033 5414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:06.856333 5414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:06.856472 5414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:06.863700 5414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:06.863849 5414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:06.864028 5414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:40:06.873970 5414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.3.keytab
--principal=kudu/127.17.205.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:06.875301 5414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:06.876636 5414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:06.889599 5420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:06.889576 5422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:06.890417 5419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:06.891283 5414 server_base.cc:1034] running on GCE node
I20260929 16:40:06.891934 5414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:06.892901 5414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:06.894104 5414 hybrid_clock.cc:648] HybridClock initialized: now 1790700006894050 us; error 46 us; skew 500 ppm
I20260929 16:40:06.894420 5414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:06 dist-test-slave-3h01 krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700006, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:06.897432 5414 init.cc:377] Logged in from keytab as kudu/127.17.205.3@KRBTEST.COM (short username kudu)
I20260929 16:40:06.898927 5414 webserver.cc:460] Webserver started at http://127.17.205.3:40787/ using document root <none> and password file <none>
I20260929 16:40:06.899670 5414 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:06.899797 5414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:06.900098 5414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:06.903077 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "ba97bc5a2b6f4acb84f8720feeeb7966"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "21ba58bbfe1396556e2af0e96fc2b54b"
server_key_iv: "8ebe93671ded9ee1e700b068819745bf"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.903965 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "ba97bc5a2b6f4acb84f8720feeeb7966"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "21ba58bbfe1396556e2af0e96fc2b54b"
server_key_iv: "8ebe93671ded9ee1e700b068819745bf"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.909308 5414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:06.912951 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.914410 5414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:40:06.914574 5414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "ba97bc5a2b6f4acb84f8720feeeb7966"
format_stamp: "Formatted at 2026-09-29 16:40:06 on dist-test-slave-3h01"
server_key: "21ba58bbfe1396556e2af0e96fc2b54b"
server_key_iv: "8ebe93671ded9ee1e700b068819745bf"
server_key_version: "encryptionkey@0"
I20260929 16:40:06.914772 5414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:06.940119 5414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:06.944541 5414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:06.944828 5414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:06.945824 5414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:06.947201 5414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:06.947299 5414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.947417 5414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:06.947468 5414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:06.979245 5414 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:41007
I20260929 16:40:06.979303 5541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:41007 every 8 connection(s)
I20260929 16:40:06.980623 5414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:40:06.989784 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5414
I20260929 16:40:06.989959 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:40:06.990370 18228 external_mini_cluster.cc:1428] Setting key 0b907291d439bc7f4400dac345e89f61
Sep 29 16:40:06 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700006, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.3@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:06.999621 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:06.983251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:41369 (local address 127.17.205.62:33643)
0929 16:40:06.983425 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:06.983429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:06.984537 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 16:40:06.986357 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:06.986362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:06.986449 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:06.986534 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:06.988082 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.988848 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.990491 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.990953 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.991063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:06.994324 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:06.994328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:06.994331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:06.994365 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:06.996688 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:06.997372 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:06.997376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:06.997378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:06.997506 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:06.997914 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:06.997917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:06.997919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:06.998189 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:06.998327 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:06.999032 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 16:40:06.999291 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:40:07.000592 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:06.983681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:33643 (local address 127.17.205.3:41369)
0929 16:40:06.984375 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:06.984426 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:06.986092 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:06.986777 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:06.986786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:06.987348 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:06.987918 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:06.987935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.988991 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.988995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:06.990356 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:06.990368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:06.991164 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:06.991171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:06.991424 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:06.992039 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:06.992061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:06.994125 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:06.996815 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:06.996824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:06.996842 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:06.997254 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:06.997603 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:06.997608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:06.997612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:06.997805 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:06.998319 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:06.998326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:06.998654 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 16:40:06.998899 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 16:40:06.999472 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":266,"threads_started":1}
I20260929 16:40:07.001987 5542 heartbeater.cc:347] Connected to a master server at 127.17.205.62:33643
I20260929 16:40:07.002264 5542 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:07.002949 5542 heartbeater.cc:511] Master 127.17.205.62:33643 requested a full tablet report, sending...
I20260929 16:40:07.003248 5275 heartbeater.cc:503] Master 127.17.205.62:33643 was elected leader, sending a full tablet report...
I20260929 16:40:07.004398 5086 ts_manager.cc:194] Registered new tserver with Master: ba97bc5a2b6f4acb84f8720feeeb7966 (127.17.205.3:41007)
I20260929 16:40:07.005414 5086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.205.3@KRBTEST.COM'} at 127.17.205.3:41369
I20260929 16:40:07.005995 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:40:07.022886 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.010466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.17.205.62:33643)
0929 16:40:07.010626 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.010632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.011048 (+ 416us) server_negotiation.cc:407] Connection header received
0929 16:40:07.011251 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.011257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.011329 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.011410 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:07.012862 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.013618 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.014925 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.015367 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.015459 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.017658 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:07.017663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:07.017667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:07.017701 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:07.020362 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.020992 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.020997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.021001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.021136 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.021479 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.021484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.021488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.021811 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:07.022031 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.022347 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.022580 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:40:07.026317 5086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:40:07.033077 5086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:40:07.068146 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.045798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.17.205.3:41007)
0929 16:40:07.060305 (+ 14507us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.060314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.060356 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:40:07.060489 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.060497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.060796 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.062251 (+ 1455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:07.062589 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.064138 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.065966 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.067018 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.067132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.067399 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.067648 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.067819 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14329,"thread_start_us":14162,"threads_started":1}
I20260929 16:40:07.068167 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.045289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:41007 (local address 127.0.0.1:36162)
0929 16:40:07.057995 (+ 12706us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:07.058052 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:07.058264 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:07.061061 (+ 2797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:07.061070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:07.061106 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:07.061545 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.061558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.064259 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.064264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.065795 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.065807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.067197 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.067233 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.067421 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.067473 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 16:40:07.067592 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:40:07.067798 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12348,"thread_start_us":13981,"threads_started":1}
I20260929 16:40:07.068930 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.045289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:36425 (local address 127.0.0.1:36608)
0929 16:40:07.059575 (+ 14286us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:07.059594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:07.059706 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:07.061667 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:07.061673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:07.061715 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:07.062173 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.062184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.063990 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.063996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.066246 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.066259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.068121 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.068127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.068313 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.068345 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:40:07.068414 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:40:07.068544 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14152,"mutex_wait_us":193,"thread_start_us":13057,"threads_started":1}
I20260929 16:40:07.070497 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.045314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.17.205.2:36425)
0929 16:40:07.060543 (+ 15229us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.060554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.060587 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:40:07.060697 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.060720 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.060990 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.061199 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:07.062398 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.063858 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.066406 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.067905 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.069746 (+ 1841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.069948 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.070016 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.070179 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15064,"thread_start_us":16093,"threads_started":1}
I20260929 16:40:07.070590 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.060375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:34881 (local address 127.0.0.1:36066)
0929 16:40:07.062486 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:07.062504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:07.062618 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:07.063577 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:07.063582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:07.063605 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:07.063912 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.063922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.065782 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.065787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.067790 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.067804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.069883 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.069890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.070048 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.070077 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:40:07.070164 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:40:07.070283 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"thread_start_us":1834,"threads_started":1}
I20260929 16:40:07.070689 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.060098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.17.205.1:34881)
0929 16:40:07.062119 (+ 2021us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.062128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.062751 (+ 623us) server_negotiation.cc:407] Connection header received
0929 16:40:07.062879 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.062887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.063135 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.063364 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:07.064068 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.065627 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.067974 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.069385 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.069518 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.069748 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.070235 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.070395 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":1738,"threads_started":1}
I20260929 16:40:07.073017 5477 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (DEFAULT_TABLE table=test-table [id=872d6722d07b41d581a063f8e7d0e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:07.074002 5344 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (DEFAULT_TABLE table=test-table [id=872d6722d07b41d581a063f8e7d0e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:07.074651 5477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf0914bf32d4b90a5cc8756f6bd0857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:07.074332 5210 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (DEFAULT_TABLE table=test-table [id=872d6722d07b41d581a063f8e7d0e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:07.075162 5344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf0914bf32d4b90a5cc8756f6bd0857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:07.075696 5210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf0914bf32d4b90a5cc8756f6bd0857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:07.086738 5560 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Bootstrap starting.
I20260929 16:40:07.089733 5562 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Bootstrap starting.
I20260929 16:40:07.090452 5560 tablet_bootstrap.cc:654] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:07.091575 5561 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Bootstrap starting.
I20260929 16:40:07.091817 5560 log.cc:824] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:07.093130 5562 tablet_bootstrap.cc:654] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:07.094399 5561 tablet_bootstrap.cc:654] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:07.094406 5562 log.cc:824] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:07.095129 5560 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: No bootstrap required, opened a new log
I20260929 16:40:07.095491 5561 log.cc:824] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:07.095695 5560 ts_tablet_manager.cc:1397] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:40:07.097875 5562 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: No bootstrap required, opened a new log
I20260929 16:40:07.098162 5562 ts_tablet_manager.cc:1397] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:40:07.098855 5561 tablet_bootstrap.cc:492] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: No bootstrap required, opened a new log
I20260929 16:40:07.099094 5561 ts_tablet_manager.cc:1397] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 16:40:07.105842 5561 raft_consensus.cc:348] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.106211 5561 raft_consensus.cc:374] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:07.105952 5560 raft_consensus.cc:348] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.106323 5561 raft_consensus.cc:729] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba97bc5a2b6f4acb84f8720feeeb7966, State: Initialized, Role: FOLLOWER
I20260929 16:40:07.106427 5560 raft_consensus.cc:374] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:07.106528 5560 raft_consensus.cc:729] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43afbe250d7241d8b5ff6090aae710e2, State: Initialized, Role: FOLLOWER
I20260929 16:40:07.106948 5561 consensus_queue.cc:260] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.107126 5560 consensus_queue.cc:260] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.108036 5542 heartbeater.cc:503] Master 127.17.205.62:33643 was elected leader, sending a full tablet report...
I20260929 16:40:07.108839 5561 ts_tablet_manager.cc:1428] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260929 16:40:07.108721 5562 raft_consensus.cc:348] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.109134 5562 raft_consensus.cc:374] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:07.109220 5562 raft_consensus.cc:729] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d3c30355df4f0d90865939f531b8f0, State: Initialized, Role: FOLLOWER
I20260929 16:40:07.109910 5562 consensus_queue.cc:260] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.112106 5562 ts_tablet_manager.cc:1428] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260929 16:40:07.113251 5409 heartbeater.cc:503] Master 127.17.205.62:33643 was elected leader, sending a full tablet report...
I20260929 16:40:07.113791 5560 ts_tablet_manager.cc:1428] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260929 16:40:07.222623 5276 tablet.cc:2369] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:07.224890 5410 tablet.cc:2369] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:07.232272 5543 tablet.cc:2369] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:07.304764 5566 raft_consensus.cc:482] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:40:07.305086 5566 raft_consensus.cc:504] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.307047 5566 leader_election.cc:288] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43afbe250d7241d8b5ff6090aae710e2 (127.17.205.2:36425), 65d3c30355df4f0d90865939f531b8f0 (127.17.205.1:34881)
I20260929 16:40:07.313328 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.307414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:41365 (local address 127.17.205.2:36425)
0929 16:40:07.307610 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.307616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.307753 (+ 137us) server_negotiation.cc:407] Connection header received
0929 16:40:07.307972 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.307976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.308034 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.308132 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:07.308532 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.309273 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.311298 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.312495 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.312612 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.312665 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.313025 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.313118 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:40:07.313472 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.307344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:36425 (local address 127.17.205.3:41365)
0929 16:40:07.307669 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:07.307684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:07.307840 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:07.308178 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:07.308182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:07.308214 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:07.308400 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.308408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.309396 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.309401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.311107 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.311121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.312628 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.312634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.312874 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.312909 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:40:07.313005 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:40:07.313138 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":17}
I20260929 16:40:07.314530 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.307506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:38143 (local address 127.17.205.1:34881)
0929 16:40:07.307733 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.307739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.308516 (+ 777us) server_negotiation.cc:407] Connection header received
0929 16:40:07.308673 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.308677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.308743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.308814 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:07.309370 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.310410 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.312538 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.313727 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.313871 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.313938 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.314224 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.314361 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:40:07.314664 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.307664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:34881 (local address 127.17.205.3:38143)
0929 16:40:07.308406 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:07.308419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:07.308539 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:07.308842 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:07.308846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:07.308869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:07.309214 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.309226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.310563 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.310566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.312391 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:07.312406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.313890 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.313896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:07.314065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.314097 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:40:07.314187 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:40:07.314325 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":19,"thread_start_us":403,"threads_started":1}
I20260929 16:40:07.314550 5364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf0914bf32d4b90a5cc8756f6bd0857" candidate_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43afbe250d7241d8b5ff6090aae710e2" is_pre_election: true
I20260929 16:40:07.315152 5364 raft_consensus.cc:2454] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba97bc5a2b6f4acb84f8720feeeb7966 in term 0.
I20260929 16:40:07.315702 5230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf0914bf32d4b90a5cc8756f6bd0857" candidate_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d3c30355df4f0d90865939f531b8f0" is_pre_election: true
I20260929 16:40:07.315898 5433 leader_election.cc:302] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43afbe250d7241d8b5ff6090aae710e2, ba97bc5a2b6f4acb84f8720feeeb7966; no voters:
I20260929 16:40:07.316171 5230 raft_consensus.cc:2454] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba97bc5a2b6f4acb84f8720feeeb7966 in term 0.
I20260929 16:40:07.316294 5566 raft_consensus.cc:2781] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:40:07.316431 5566 raft_consensus.cc:482] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:40:07.316532 5566 raft_consensus.cc:3037] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:07.318715 5566 raft_consensus.cc:504] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.319340 5566 leader_election.cc:288] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [CANDIDATE]: Term 1 election: Requested vote from peers 43afbe250d7241d8b5ff6090aae710e2 (127.17.205.2:36425), 65d3c30355df4f0d90865939f531b8f0 (127.17.205.1:34881)
I20260929 16:40:07.319769 5364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf0914bf32d4b90a5cc8756f6bd0857" candidate_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43afbe250d7241d8b5ff6090aae710e2"
I20260929 16:40:07.319799 5230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf0914bf32d4b90a5cc8756f6bd0857" candidate_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d3c30355df4f0d90865939f531b8f0"
I20260929 16:40:07.319963 5364 raft_consensus.cc:3037] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:07.320000 5230 raft_consensus.cc:3037] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:07.322132 5230 raft_consensus.cc:2454] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba97bc5a2b6f4acb84f8720feeeb7966 in term 1.
I20260929 16:40:07.322142 5364 raft_consensus.cc:2454] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba97bc5a2b6f4acb84f8720feeeb7966 in term 1.
I20260929 16:40:07.322532 5431 leader_election.cc:302] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43afbe250d7241d8b5ff6090aae710e2, 65d3c30355df4f0d90865939f531b8f0, ba97bc5a2b6f4acb84f8720feeeb7966; no voters:
I20260929 16:40:07.322867 5566 raft_consensus.cc:2781] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:07.323342 5566 raft_consensus.cc:686] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 1 LEADER]: Becoming Leader. State: Replica: ba97bc5a2b6f4acb84f8720feeeb7966, State: Running, Role: LEADER
I20260929 16:40:07.323872 5566 consensus_queue.cc:237] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } }
I20260929 16:40:07.328197 5086 catalog_manager.cc:5495] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba97bc5a2b6f4acb84f8720feeeb7966 (127.17.205.3). New cstate: current_term: 1 leader_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba97bc5a2b6f4acb84f8720feeeb7966" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 } health_report { overall_health: UNKNOWN } } }
I20260929 16:40:07.353317 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.347323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.17.205.3:41007)
0929 16:40:07.347496 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.347501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.347725 (+ 224us) server_negotiation.cc:407] Connection header received
0929 16:40:07.347937 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.347941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.348019 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.348142 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:07.348516 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.349193 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.351140 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.351619 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.351720 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.352250 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:07.352788 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:07.352914 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.353033 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.353106 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:40:07.361364 5364 raft_consensus.cc:1261] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 1 FOLLOWER]: Refusing update from remote peer ba97bc5a2b6f4acb84f8720feeeb7966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:07.361364 5230 raft_consensus.cc:1261] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 1 FOLLOWER]: Refusing update from remote peer ba97bc5a2b6f4acb84f8720feeeb7966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:07.362290 5570 consensus_queue.cc:1035] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d3c30355df4f0d90865939f531b8f0" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 34881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:07.362617 5566 consensus_queue.cc:1035] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43afbe250d7241d8b5ff6090aae710e2" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:07.377491 5574 mvcc.cc:205] Tried to move back new op lower bound from 7334707230142406656 to 7334707230002286592. Current Snapshot: MvccSnapshot[applied={T|T < 7334707230142406656}]
I20260929 16:40:07.386018 5085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:07.386348 5085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:07.390213 5085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41af2aac424a4c83909618ece4545c9f: Sending DeleteTablet for 3 replicas of tablet 0cf0914bf32d4b90a5cc8756f6bd0857
I20260929 16:40:07.391460 5344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:36608
I20260929 16:40:07.391546 5477 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:36162
I20260929 16:40:07.391798 5210 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf0914bf32d4b90a5cc8756f6bd0857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:07 UTC) from {username='kudu', principal='kudu/127.17.205.62@KRBTEST.COM'} at 127.0.0.1:36066
I20260929 16:40:07.393558 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5147
I20260929 16:40:07.402974 5583 tablet_replica.cc:330] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: stopping tablet replica
I20260929 16:40:07.404260 5583 raft_consensus.cc:2229] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:07.405561 5583 raft_consensus.cc:2258] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:07.408838 5583 ts_tablet_manager.cc:1905] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:07.412964 5585 tablet_replica.cc:330] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: stopping tablet replica
I20260929 16:40:07.413129 5584 tablet_replica.cc:330] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: stopping tablet replica
I20260929 16:40:07.413447 5585 raft_consensus.cc:2229] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:07.413491 5584 raft_consensus.cc:2229] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:40:07.414191 5584 raft_consensus.cc:2258] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:07.415614 5585 raft_consensus.cc:2258] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 16:40:07 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.1@KRBTEST.COM
I20260929 16:40:07.420642 5585 ts_tablet_manager.cc:1905] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:07.420778 5584 ts_tablet_manager.cc:1905] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:07.427690 5585 ts_tablet_manager.cc:1918] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:07.427696 5584 ts_tablet_manager.cc:1918] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:07.427867 5584 log.cc:1196] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/0cf0914bf32d4b90a5cc8756f6bd0857
I20260929 16:40:07.427956 5585 log.cc:1196] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/0cf0914bf32d4b90a5cc8756f6bd0857
I20260929 16:40:07.428494 5585 ts_tablet_manager.cc:1939] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 65d3c30355df4f0d90865939f531b8f0: Deleting consensus metadata
I20260929 16:40:07.429476 5583 ts_tablet_manager.cc:1918] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:07.429721 5583 log.cc:1196] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/0cf0914bf32d4b90a5cc8756f6bd0857
I20260929 16:40:07.430416 5071 catalog_manager.cc:4841] TS 65d3c30355df4f0d90865939f531b8f0 (127.17.205.1:34881): tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (table test-table [id=872d6722d07b41d581a063f8e7d0e24e]) successfully deleted
I20260929 16:40:07.430831 5583 ts_tablet_manager.cc:1939] T 0cf0914bf32d4b90a5cc8756f6bd0857 P 43afbe250d7241d8b5ff6090aae710e2: Deleting consensus metadata
I20260929 16:40:07.431742 5584 ts_tablet_manager.cc:1939] T 0cf0914bf32d4b90a5cc8756f6bd0857 P ba97bc5a2b6f4acb84f8720feeeb7966: Deleting consensus metadata
I20260929 16:40:07.432375 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.395559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.17.205.1:34881)
0929 16:40:07.395758 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.395765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.407945 (+ 12180us) server_negotiation.cc:407] Connection header received
0929 16:40:07.408450 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.408456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.408552 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.408625 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:07.410496 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.411716 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.414845 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.415592 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.415752 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.424513 (+ 8761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:07.424529 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:07.424548 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:07.424611 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:07.428772 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.429759 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.429770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.429774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.429954 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.430438 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.430442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.430446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.430886 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:07.431165 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.431784 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.432009 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:40:07.432726 5073 catalog_manager.cc:4841] TS 43afbe250d7241d8b5ff6090aae710e2 (127.17.205.2:36425): tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (table test-table [id=872d6722d07b41d581a063f8e7d0e24e]) successfully deleted
I20260929 16:40:07.433657 5071 catalog_manager.cc:4841] TS ba97bc5a2b6f4acb84f8720feeeb7966 (127.17.205.3:41007): tablet 0cf0914bf32d4b90a5cc8756f6bd0857 (table test-table [id=872d6722d07b41d581a063f8e7d0e24e]) successfully deleted
I20260929 16:40:07.433846 5270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:07.521936 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5147
I20260929 16:40:07.543061 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5281
Sep 29 16:40:07 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.2@KRBTEST.COM
I20260929 16:40:07.560159 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.544084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.17.205.2:36425)
0929 16:40:07.544250 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.544254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.544273 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:40:07.544350 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.544354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.544441 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.544523 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:07.546164 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.546908 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.548830 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.549231 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.549327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.554317 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:07.554333 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:07.554351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:07.554394 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:07.557365 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.558161 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.558169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.558171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.558311 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.558664 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.558667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.558669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.559056 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:07.559339 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.559676 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.559860 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:40:07.560851 5404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:07.650077 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5281
I20260929 16:40:07.670959 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5414
Sep 29 16:40:07 dist-test-slave-3h01 krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.3@KRBTEST.COM
I20260929 16:40:07.689519 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:07.672106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.17.205.3:41007)
0929 16:40:07.672365 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:07.672374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:07.672650 (+ 276us) server_negotiation.cc:407] Connection header received
0929 16:40:07.672848 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:07.672852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:07.672975 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:07.673081 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:07.674835 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.675757 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.677602 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:07.678064 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:07.678165 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:07.683591 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:07.683608 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:07.683624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:07.683681 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:07.686594 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.687292 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.687300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.687302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.687460 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:07.687835 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:07.687838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:07.687840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:07.688201 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:07.688573 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:07.688848 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:40:07.689083 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 16:40:07.690356 5537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:07.811158 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5414
I20260929 16:40:07.827042 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5054
I20260929 16:40:07.828236 5116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:07.937183 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5054
2026-09-29T16:40:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:07Z Disabled control of system clock
I20260929 16:40:08.007665 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37251
--webserver_interface=127.17.205.62
--webserver_port=0
--builtin_ntp_servers=127.17.205.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:37251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:40:08.340387 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:08.340683 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:08.340781 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:08.348192 5594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:08.348315 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:08.348351 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:08.348408 5594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:08.348467 5594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:08.358208 5594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:37251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:08.359596 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:08.361383 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:08.372798 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:08.373000 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:08.372774 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:08.373584 5594 server_base.cc:1034] running on GCE node
I20260929 16:40:08.374326 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:08.375770 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:08.376971 5594 hybrid_clock.cc:648] HybridClock initialized: now 1790700008376908 us; error 50 us; skew 500 ppm
I20260929 16:40:08.377307 5594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:08.379495 5594 webserver.cc:460] Webserver started at http://127.17.205.62:42211/ using document root <none> and password file <none>
I20260929 16:40:08.380172 5594 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:08.380263 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:08.380559 5594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:08.383455 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "ab3d5d25bd694c888ab3762918bb5c4d"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "3f3526e6e3a4ec81fb24d05b40cb2b44"
server_key_iv: "4b9d3e3f37133e4dfca3316cea31d739"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.384241 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "ab3d5d25bd694c888ab3762918bb5c4d"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "3f3526e6e3a4ec81fb24d05b40cb2b44"
server_key_iv: "4b9d3e3f37133e4dfca3316cea31d739"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.389210 5594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:40:08.392578 5608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:08.393882 5594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:08.394048 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "ab3d5d25bd694c888ab3762918bb5c4d"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "3f3526e6e3a4ec81fb24d05b40cb2b44"
server_key_iv: "4b9d3e3f37133e4dfca3316cea31d739"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.394241 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:08.406129 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:08.406953 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:08.407235 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:08.423763 5594 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:37251
I20260929 16:40:08.423795 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:37251 every 8 connection(s)
I20260929 16:40:08.425051 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:08.428790 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:08.430080 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5594
I20260929 16:40:08.430279 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:08.430851 18228 external_mini_cluster.cc:1428] Setting key 151f0cccc98ec6abd10efa716ae1016e
I20260929 16:40:08.441915 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: Bootstrap starting.
I20260929 16:40:08.444321 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:08.432921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.17.205.62:37251)
0929 16:40:08.434122 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:08.434138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:08.434200 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:40:08.435910 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:08.435940 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:08.436443 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:08.436816 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:08.437358 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.438585 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.439914 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.440579 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.440751 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:08.441877 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:08.441885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:08.441902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:08.442468 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:08.442549 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:08.442570 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:08.442668 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:08.443022 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:40:08.443363 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":367,"threads_started":1}
I20260929 16:40:08.445180 5660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:08.446178 5660 log.cc:824] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:08.449011 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: No bootstrap required, opened a new log
I20260929 16:40:08.456259 5660 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } }
I20260929 16:40:08.456615 5660 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:08.456702 5660 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3d5d25bd694c888ab3762918bb5c4d, State: Initialized, Role: FOLLOWER
I20260929 16:40:08.457263 5660 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } }
I20260929 16:40:08.457484 5660 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:08.457604 5660 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:08.457762 5660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:08.460126 5660 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } }
I20260929 16:40:08.460608 5660 leader_election.cc:302] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3d5d25bd694c888ab3762918bb5c4d; no voters:
I20260929 16:40:08.461184 5660 leader_election.cc:288] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:08.461329 5665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:08.462047 5665 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [term 1 LEADER]: Becoming Leader. State: Replica: ab3d5d25bd694c888ab3762918bb5c4d, State: Running, Role: LEADER
I20260929 16:40:08.462695 5665 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } }
I20260929 16:40:08.463325 5660 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:08.465574 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3d5d25bd694c888ab3762918bb5c4d. Latest consensus state: current_term: 1 leader_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } } }
I20260929 16:40:08.466080 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:08.465548 5666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3d5d25bd694c888ab3762918bb5c4d" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 37251 } } }
I20260929 16:40:08.466535 5666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:08.468206 5673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:08.473237 5673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:08.482426 5673 catalog_manager.cc:1347] Generated new cluster ID: a089bcc963f34130adbc46411a793bfa
I20260929 16:40:08.482545 5673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:08.494596 5673 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:08.495996 5673 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:08.505478 5673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: Generated new TSK 0
I20260929 16:40:08.506316 5673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:08.517867 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.1:0
--local_ip_for_outbound_sockets=127.17.205.1
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--builtin_ntp_servers=127.17.205.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:40:08.849678 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:08.849962 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:08.850049 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:08.857301 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:08.857419 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:08.857525 5684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.1
I20260929 16:40:08.866864 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.205.1
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.1
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:08.868130 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:08.869863 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:08.883188 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:08.883246 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:08.883150 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:08.883813 5684 server_base.cc:1034] running on GCE node
I20260929 16:40:08.884511 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:08.885505 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:08.886745 5684 hybrid_clock.cc:648] HybridClock initialized: now 1790700008886683 us; error 56 us; skew 500 ppm
I20260929 16:40:08.887095 5684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:08.889391 5684 webserver.cc:460] Webserver started at http://127.17.205.1:38805/ using document root <none> and password file <none>
I20260929 16:40:08.890093 5684 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:08.890195 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:08.890511 5684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:08.893455 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data/instance:
uuid: "20420b836a764dfd8774e063fe37be09"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "717f294f3017e4c04679b333fcd03f6e"
server_key_iv: "88dca7a26ef9240a5ecee8b1cb21b67d"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.894243 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance:
uuid: "20420b836a764dfd8774e063fe37be09"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "717f294f3017e4c04679b333fcd03f6e"
server_key_iv: "88dca7a26ef9240a5ecee8b1cb21b67d"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.899663 5684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:40:08.903187 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:08.904633 5684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:08.904801 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal
uuid: "20420b836a764dfd8774e063fe37be09"
format_stamp: "Formatted at 2026-09-29 16:40:08 on dist-test-slave-3h01"
server_key: "717f294f3017e4c04679b333fcd03f6e"
server_key_iv: "88dca7a26ef9240a5ecee8b1cb21b67d"
server_key_version: "encryptionkey@0"
I20260929 16:40:08.905006 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:08.918002 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:08.918860 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:08.919145 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:08.920161 5684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:08.921640 5684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:08.921734 5684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:08.921844 5684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:08.921917 5684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:08.951911 5684 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.1:36713
I20260929 16:40:08.951963 5810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.1:36713 every 8 connection(s)
I20260929 16:40:08.953217 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/data/info.pb
I20260929 16:40:08.961529 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5684
I20260929 16:40:08.961699 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal/instance
I20260929 16:40:08.962102 18228 external_mini_cluster.cc:1428] Setting key 5b5503651a3dceea6c539919d6fa1544
I20260929 16:40:08.964929 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.2:0
--local_ip_for_outbound_sockets=127.17.205.2
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--builtin_ntp_servers=127.17.205.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:40:08.966050 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:08.955788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.1:46569 (local address 127.17.205.62:37251)
0929 16:40:08.956273 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:08.956287 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:08.957520 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 16:40:08.959269 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:08.959273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:08.959354 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:08.959453 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:08.960545 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.961335 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.963199 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.963693 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.963815 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:08.965162 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:08.965177 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:08.965181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:08.965205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:08.965258 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:08.965275 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:08.965371 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:08.965556 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:40:08.965662 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260929 16:40:08.966962 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:08.956447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37251 (local address 127.17.205.1:46569)
0929 16:40:08.957355 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:08.957399 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:08.958989 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:08.959655 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:08.959664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:08.959777 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:08.960359 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:08.960376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.961516 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.961520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:08.963024 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:08.963039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:08.963909 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:08.963917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:08.964145 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:08.964823 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:08.964826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:08.964867 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:08.964950 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:08.965365 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:08.965374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:08.965380 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:40:08.965524 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:40:08.965845 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":266,"threads_started":1}
I20260929 16:40:08.968544 5811 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37251
I20260929 16:40:08.968847 5811 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:08.969581 5811 heartbeater.cc:511] Master 127.17.205.62:37251 requested a full tablet report, sending...
I20260929 16:40:08.971642 5625 ts_manager.cc:194] Registered new tserver with Master: 20420b836a764dfd8774e063fe37be09 (127.17.205.1:36713)
I20260929 16:40:08.973105 5625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.1:46569
W20260929 16:40:09.300983 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:09.301291 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:09.301416 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:09.308956 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:09.309079 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:09.309247 5815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.2
I20260929 16:40:09.318656 5815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.205.2
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.2
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:09.320123 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:09.321967 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:09.334852 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:09.334939 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:09.334909 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:09.335717 5815 server_base.cc:1034] running on GCE node
I20260929 16:40:09.336293 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:09.337265 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:09.338469 5815 hybrid_clock.cc:648] HybridClock initialized: now 1790700009338420 us; error 44 us; skew 500 ppm
I20260929 16:40:09.338819 5815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:09.341120 5815 webserver.cc:460] Webserver started at http://127.17.205.2:46759/ using document root <none> and password file <none>
I20260929 16:40:09.341807 5815 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:09.341910 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:09.342226 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:09.345264 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data/instance:
uuid: "e65cfea3c1d748b5987b8d52c4f0629e"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "300145f19fec002ea25aab5555db3ed2"
server_key_iv: "a760fe062c0b378c5bbe858bd43f8902"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.346096 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance:
uuid: "e65cfea3c1d748b5987b8d52c4f0629e"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "300145f19fec002ea25aab5555db3ed2"
server_key_iv: "a760fe062c0b378c5bbe858bd43f8902"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.351599 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:40:09.355363 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.356701 5815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:09.356879 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal
uuid: "e65cfea3c1d748b5987b8d52c4f0629e"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "300145f19fec002ea25aab5555db3ed2"
server_key_iv: "a760fe062c0b378c5bbe858bd43f8902"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.357085 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:09.383814 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:09.384677 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:09.385002 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:09.386014 5815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:09.387506 5815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:09.387603 5815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.387705 5815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:09.387775 5815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.417234 5815 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.2:35449
I20260929 16:40:09.417301 5941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.2:35449 every 8 connection(s)
I20260929 16:40:09.418506 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/data/info.pb
I20260929 16:40:09.421573 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5815
I20260929 16:40:09.421713 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal/instance
I20260929 16:40:09.422253 18228 external_mini_cluster.cc:1428] Setting key 1a2b6fdbb5c62a048870817f7ff114f8
I20260929 16:40:09.427129 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.205.3:0
--local_ip_for_outbound_sockets=127.17.205.3
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--builtin_ntp_servers=127.17.205.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:40:09.432904 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.420894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:39869 (local address 127.17.205.62:37251)
0929 16:40:09.421111 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.421117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.422749 (+ 1632us) server_negotiation.cc:407] Connection header received
0929 16:40:09.425137 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.425141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.425209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.425301 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:09.426498 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.427650 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.429815 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.430465 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.430587 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.432103 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:09.432110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:09.432115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:09.432136 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:09.432174 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:09.432189 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:09.432270 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.432486 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.432580 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:40:09.434350 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.421487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37251 (local address 127.17.205.2:39869)
0929 16:40:09.422577 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:09.422626 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:09.424860 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:09.425553 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:09.425561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:09.425658 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:09.426321 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.426349 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.427839 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.427845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.429670 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.429685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.430683 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.430689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.430957 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.431749 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:09.431753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:09.431804 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:09.431908 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:09.432278 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:09.432295 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:09.432303 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:40:09.432476 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:40:09.432870 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":313,"threads_started":1}
I20260929 16:40:09.435856 5942 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37251
I20260929 16:40:09.436197 5942 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:09.437072 5942 heartbeater.cc:511] Master 127.17.205.62:37251 requested a full tablet report, sending...
I20260929 16:40:09.438524 5625 ts_manager.cc:194] Registered new tserver with Master: e65cfea3c1d748b5987b8d52c4f0629e (127.17.205.2:35449)
I20260929 16:40:09.439388 5625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.2:39869
W20260929 16:40:09.752024 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:09.752317 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:09.752406 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:09.759706 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:09.759825 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:09.759955 5945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.205.3
I20260929 16:40:09.769423 5945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.205.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.205.3
--webserver_port=0
--tserver_master_addrs=127.17.205.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.205.3
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:09.770761 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:09.772617 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:09.785413 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:09.785517 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:09.785415 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:09.786347 5945 server_base.cc:1034] running on GCE node
I20260929 16:40:09.786990 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:09.788009 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:09.789203 5945 hybrid_clock.cc:648] HybridClock initialized: now 1790700009789140 us; error 53 us; skew 500 ppm
I20260929 16:40:09.789526 5945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:40:09.791837 5945 webserver.cc:460] Webserver started at http://127.17.205.3:38959/ using document root <none> and password file <none>
I20260929 16:40:09.792541 5945 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:09.792635 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:09.792940 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:09.795861 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data/instance:
uuid: "7a69826257cc4cb38e3b43aa9c00e394"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "cf093a5841506d4a2b5e7118da57bef4"
server_key_iv: "2b63f767c5bb4a7b456c34401b88606c"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.796624 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance:
uuid: "7a69826257cc4cb38e3b43aa9c00e394"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "cf093a5841506d4a2b5e7118da57bef4"
server_key_iv: "2b63f767c5bb4a7b456c34401b88606c"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.801676 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:09.805148 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.806499 5945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:40:09.806653 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal
uuid: "7a69826257cc4cb38e3b43aa9c00e394"
format_stamp: "Formatted at 2026-09-29 16:40:09 on dist-test-slave-3h01"
server_key: "cf093a5841506d4a2b5e7118da57bef4"
server_key_iv: "2b63f767c5bb4a7b456c34401b88606c"
server_key_version: "encryptionkey@0"
I20260929 16:40:09.806859 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:09.834846 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:09.835773 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:09.836059 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:09.837064 5945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:40:09.838567 5945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:40:09.838665 5945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.838781 5945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:40:09.838851 5945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:09.868984 5945 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.3:34223
I20260929 16:40:09.869038 6071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.3:34223 every 8 connection(s)
I20260929 16:40:09.870275 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/data/info.pb
I20260929 16:40:09.875156 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 5945
I20260929 16:40:09.875377 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal/instance
I20260929 16:40:09.875897 18228 external_mini_cluster.cc:1428] Setting key e52310726b7a476001745b32f07d94de
I20260929 16:40:09.882858 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.872764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.3:43507 (local address 127.17.205.62:37251)
0929 16:40:09.872965 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.872969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.874379 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 16:40:09.876654 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.876661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.876749 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.876850 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:09.878071 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.878979 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.880542 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.880931 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.881026 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.882241 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:09.882245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:09.882248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:09.882264 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:09.882298 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:09.882309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:09.882370 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.882549 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.882623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:40:09.883912 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.873344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:37251 (local address 127.17.205.3:43507)
0929 16:40:09.874221 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:09.874263 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:09.876397 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:09.877142 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:09.877151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:09.877256 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:09.877884 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.877910 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.879140 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.879144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.880423 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.880436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.881102 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.881106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.881316 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.881964 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:40:09.881967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:09.882012 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:40:09.882094 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:09.882374 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:09.882381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:09.882386 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:40:09.882540 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:40:09.882851 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":238,"threads_started":1}
I20260929 16:40:09.885164 6072 heartbeater.cc:347] Connected to a master server at 127.17.205.62:37251
I20260929 16:40:09.885439 6072 heartbeater.cc:464] Registering TS with master...
I20260929 16:40:09.886171 6072 heartbeater.cc:511] Master 127.17.205.62:37251 requested a full tablet report, sending...
I20260929 16:40:09.887521 5625 ts_manager.cc:194] Registered new tserver with Master: 7a69826257cc4cb38e3b43aa9c00e394 (127.17.205.3:34223)
I20260929 16:40:09.888269 5625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.205.3:43507
I20260929 16:40:09.890880 18228 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:40:09.899947 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.894705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.17.205.62:37251)
0929 16:40:09.894893 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.894897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.895305 (+ 408us) server_negotiation.cc:407] Connection header received
0929 16:40:09.895440 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.895444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.895515 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.895597 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:09.895998 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.896657 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.897937 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.898362 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.898454 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.899429 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:09.899434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:09.899437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:09.899453 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:09.899483 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:09.899494 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:09.899561 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.899634 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.899704 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:40:09.903411 5625 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:40:09.910110 5625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:40:09.932174 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.921375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.2:35449 (local address 127.0.0.1:60054)
0929 16:40:09.922869 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:09.922928 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:09.923105 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:09.924998 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:09.925005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:09.925034 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:09.925418 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.925431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.927422 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.927428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.929775 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.929788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.931184 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.931190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.931429 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.931485 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:40:09.931614 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:40:09.931808 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":63,"thread_start_us":3284,"threads_started":1}
I20260929 16:40:09.932209 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.921680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.17.205.2:35449)
0929 16:40:09.923648 (+ 1968us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.923656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.923683 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:40:09.923789 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.923796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.924772 (+ 976us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.924960 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:09.925789 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.927303 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.929938 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.931010 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.931124 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.931443 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.931677 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.931852 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":1707,"threads_started":1}
I20260929 16:40:09.932587 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.921554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36713 (local address 127.0.0.1:34592)
0929 16:40:09.924882 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:09.924902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:09.925117 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:09.926002 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:09.926007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:09.926033 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:09.926378 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.926389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.927842 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.927847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.929653 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.929670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.931860 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.931865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.932029 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.932061 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:40:09.932148 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:40:09.932275 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3179,"mutex_wait_us":1,"thread_start_us":871,"threads_started":1}
I20260929 16:40:09.933434 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.922053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.17.205.1:36713)
0929 16:40:09.924451 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.924459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.925003 (+ 544us) server_negotiation.cc:407] Connection header received
0929 16:40:09.925300 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.925309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.925550 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.925707 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:09.926531 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.927728 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.929899 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.931361 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.931470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.931687 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.932972 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.933145 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2239,"thread_start_us":2138,"threads_started":1}
I20260929 16:40:09.933836 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.924257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:34223 (local address 127.0.0.1:58762)
0929 16:40:09.925406 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:09.925422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:09.925524 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:09.927034 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:09.927039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:09.927063 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:09.927417 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.927429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.929290 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.929296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.931384 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:09.931397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.933016 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.933022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:09.933177 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.933209 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:40:09.933277 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:40:09.933393 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":1326,"threads_started":1}
I20260929 16:40:09.934762 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:09.924521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.17.205.3:34223)
0929 16:40:09.926523 (+ 2002us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:09.926531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:09.926560 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:40:09.926640 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:09.926647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:09.926856 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:09.927022 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:09.927627 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.929145 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.931544 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:09.932846 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:09.933947 (+ 1101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:09.934212 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:09.934289 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:40:09.934451 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":1691,"threads_started":1}
I20260929 16:40:09.936547 5877 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 (DEFAULT_TABLE table=test-table [id=eab4e7b60be549e49c72696ba3491412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:09.936776 5746 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 (DEFAULT_TABLE table=test-table [id=eab4e7b60be549e49c72696ba3491412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:09.937692 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe09536d5064d0281a0d2e6db07cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:09.938128 5746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe09536d5064d0281a0d2e6db07cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:09.938474 6007 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 (DEFAULT_TABLE table=test-table [id=eab4e7b60be549e49c72696ba3491412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:40:09.939833 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe09536d5064d0281a0d2e6db07cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:09.949882 6090 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Bootstrap starting.
I20260929 16:40:09.952217 6091 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Bootstrap starting.
I20260929 16:40:09.953504 6090 tablet_bootstrap.cc:654] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:09.954746 6090 log.cc:824] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:09.955847 6091 tablet_bootstrap.cc:654] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:09.957093 6091 log.cc:824] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:09.958844 6090 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: No bootstrap required, opened a new log
I20260929 16:40:09.959192 6090 ts_tablet_manager.cc:1397] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:40:09.968681 6094 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Bootstrap starting.
I20260929 16:40:09.969853 6091 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: No bootstrap required, opened a new log
I20260929 16:40:09.970139 6091 ts_tablet_manager.cc:1397] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260929 16:40:09.970036 6090 raft_consensus.cc:348] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.970714 6090 raft_consensus.cc:374] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:09.970815 6090 raft_consensus.cc:729] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65cfea3c1d748b5987b8d52c4f0629e, State: Initialized, Role: FOLLOWER
I20260929 16:40:09.971431 6090 consensus_queue.cc:260] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.972404 6094 tablet_bootstrap.cc:654] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:09.972470 5942 heartbeater.cc:503] Master 127.17.205.62:37251 was elected leader, sending a full tablet report...
I20260929 16:40:09.973074 6090 ts_tablet_manager.cc:1428] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 16:40:09.973623 6094 log.cc:824] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:09.976877 5811 heartbeater.cc:503] Master 127.17.205.62:37251 was elected leader, sending a full tablet report...
I20260929 16:40:09.977000 6094 tablet_bootstrap.cc:492] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: No bootstrap required, opened a new log
I20260929 16:40:09.977283 6094 ts_tablet_manager.cc:1397] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:40:09.980688 6091 raft_consensus.cc:348] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.981137 6091 raft_consensus.cc:374] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:09.981236 6091 raft_consensus.cc:729] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20420b836a764dfd8774e063fe37be09, State: Initialized, Role: FOLLOWER
I20260929 16:40:09.981851 6091 consensus_queue.cc:260] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.983544 6091 ts_tablet_manager.cc:1428] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
I20260929 16:40:09.986076 6094 raft_consensus.cc:348] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.986409 6094 raft_consensus.cc:374] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:09.986498 6094 raft_consensus.cc:729] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a69826257cc4cb38e3b43aa9c00e394, State: Initialized, Role: FOLLOWER
I20260929 16:40:09.986987 6094 consensus_queue.cc:260] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:09.987918 6072 heartbeater.cc:503] Master 127.17.205.62:37251 was elected leader, sending a full tablet report...
I20260929 16:40:09.988530 6094 ts_tablet_manager.cc:1428] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260929 16:40:10.121955 6073 tablet.cc:2369] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:10.170413 5943 tablet.cc:2369] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:40:10.205570 5812 tablet.cc:2369] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:10.234174 6095 raft_consensus.cc:482] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:40:10.234462 6095 raft_consensus.cc:504] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:10.236125 6095 leader_election.cc:288] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20420b836a764dfd8774e063fe37be09 (127.17.205.1:36713), 7a69826257cc4cb38e3b43aa9c00e394 (127.17.205.3:34223)
I20260929 16:40:10.242780 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.236696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:43713 (local address 127.17.205.3:34223)
0929 16:40:10.236926 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.236933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.237557 (+ 624us) server_negotiation.cc:407] Connection header received
0929 16:40:10.237719 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.237726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.237803 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.237874 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:10.238491 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.239204 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.241159 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.241911 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.242060 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.242127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.242395 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.242532 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:40:10.242798 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.236653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.205.2:57131 (local address 127.17.205.1:36713)
0929 16:40:10.236903 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.236909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.237664 (+ 755us) server_negotiation.cc:407] Connection header received
0929 16:40:10.237819 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.237823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.237907 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.237998 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:40:10.238550 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.239202 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.241143 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.241884 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.242010 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.242081 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.242406 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.242544 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:40:10.242902 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.236387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.1:36713 (local address 127.17.205.2:57131)
0929 16:40:10.237534 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:10.237551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:10.237691 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:10.238020 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:10.238023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:10.238054 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:10.238401 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:10.238411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.239350 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.239356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:10.240946 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:10.240958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.242075 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.242081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:10.242257 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.242279 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:40:10.242355 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:40:10.242532 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":170,"thread_start_us":708,"threads_started":1}
I20260929 16:40:10.243012 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.236387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.3:34223 (local address 127.17.205.2:43713)
0929 16:40:10.237442 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:10.237457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:10.237583 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:10.237910 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:10.237915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:10.237936 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:40:10.238341 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:10.238351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.239360 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.239364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:10.240961 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:10.240973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.242090 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.242096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:10.242243 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.242267 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:40:10.242343 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:40:10.242523 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":88,"thread_start_us":707,"threads_started":1}
I20260929 16:40:10.244017 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe09536d5064d0281a0d2e6db07cc20" candidate_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20420b836a764dfd8774e063fe37be09" is_pre_election: true
I20260929 16:40:10.244104 6027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe09536d5064d0281a0d2e6db07cc20" candidate_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69826257cc4cb38e3b43aa9c00e394" is_pre_election: true
I20260929 16:40:10.244499 5766 raft_consensus.cc:2454] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65cfea3c1d748b5987b8d52c4f0629e in term 0.
I20260929 16:40:10.244545 6027 raft_consensus.cc:2454] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65cfea3c1d748b5987b8d52c4f0629e in term 0.
I20260929 16:40:10.245103 5831 leader_election.cc:302] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20420b836a764dfd8774e063fe37be09, e65cfea3c1d748b5987b8d52c4f0629e; no voters:
I20260929 16:40:10.245434 6095 raft_consensus.cc:2781] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:40:10.245558 6095 raft_consensus.cc:482] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:40:10.245644 6095 raft_consensus.cc:3037] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:10.247840 6095 raft_consensus.cc:504] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:10.248392 6095 leader_election.cc:288] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [CANDIDATE]: Term 1 election: Requested vote from peers 20420b836a764dfd8774e063fe37be09 (127.17.205.1:36713), 7a69826257cc4cb38e3b43aa9c00e394 (127.17.205.3:34223)
I20260929 16:40:10.248823 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe09536d5064d0281a0d2e6db07cc20" candidate_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20420b836a764dfd8774e063fe37be09"
I20260929 16:40:10.248908 6027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe09536d5064d0281a0d2e6db07cc20" candidate_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69826257cc4cb38e3b43aa9c00e394"
I20260929 16:40:10.248996 5766 raft_consensus.cc:3037] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:10.249109 6027 raft_consensus.cc:3037] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:10.251358 6027 raft_consensus.cc:2454] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65cfea3c1d748b5987b8d52c4f0629e in term 1.
I20260929 16:40:10.251359 5766 raft_consensus.cc:2454] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65cfea3c1d748b5987b8d52c4f0629e in term 1.
I20260929 16:40:10.251765 5833 leader_election.cc:302] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a69826257cc4cb38e3b43aa9c00e394, e65cfea3c1d748b5987b8d52c4f0629e; no voters:
I20260929 16:40:10.252048 6095 raft_consensus.cc:2781] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:10.252466 6095 raft_consensus.cc:686] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 1 LEADER]: Becoming Leader. State: Replica: e65cfea3c1d748b5987b8d52c4f0629e, State: Running, Role: LEADER
I20260929 16:40:10.252965 6095 consensus_queue.cc:237] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } }
I20260929 16:40:10.257334 5624 catalog_manager.cc:5495] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e reported cstate change: term changed from 0 to 1, leader changed from <none> to e65cfea3c1d748b5987b8d52c4f0629e (127.17.205.2). New cstate: current_term: 1 leader_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65cfea3c1d748b5987b8d52c4f0629e" member_type: VOTER last_known_addr { host: "127.17.205.2" port: 35449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 } health_report { overall_health: UNKNOWN } } }
I20260929 16:40:10.301400 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.296212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.17.205.2:35449)
0929 16:40:10.296401 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.296407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.296459 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:40:10.296684 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.296688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.296790 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.296917 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:10.297293 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.298033 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.299374 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.299775 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.299886 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.300354 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:10.300871 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:10.301012 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.301084 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.301163 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:40:10.309543 6027 raft_consensus.cc:1261] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 1 FOLLOWER]: Refusing update from remote peer e65cfea3c1d748b5987b8d52c4f0629e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:10.309543 5766 raft_consensus.cc:1261] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 1 FOLLOWER]: Refusing update from remote peer e65cfea3c1d748b5987b8d52c4f0629e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:10.310513 6101 consensus_queue.cc:1035] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a69826257cc4cb38e3b43aa9c00e394" member_type: VOTER last_known_addr { host: "127.17.205.3" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:10.310881 6095 consensus_queue.cc:1035] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20420b836a764dfd8774e063fe37be09" member_type: VOTER last_known_addr { host: "127.17.205.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:10.322072 6104 mvcc.cc:205] Tried to move back new op lower bound from 7334707242217963520 to 7334707241999859712. Current Snapshot: MvccSnapshot[applied={T|T < 7334707242217963520}]
I20260929 16:40:10.324405 6106 mvcc.cc:205] Tried to move back new op lower bound from 7334707242217963520 to 7334707241999859712. Current Snapshot: MvccSnapshot[applied={T|T < 7334707242217963520}]
I20260929 16:40:10.335979 5624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:10.336293 5624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:40:10.340034 5624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab3d5d25bd694c888ab3762918bb5c4d: Sending DeleteTablet for 3 replicas of tablet 8fe09536d5064d0281a0d2e6db07cc20
I20260929 16:40:10.341171 5877 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:10 UTC) from {username='slave'} at 127.0.0.1:60054
I20260929 16:40:10.341167 5746 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:10 UTC) from {username='slave'} at 127.0.0.1:34592
I20260929 16:40:10.341167 6007 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe09536d5064d0281a0d2e6db07cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:40:10 UTC) from {username='slave'} at 127.0.0.1:58762
I20260929 16:40:10.343717 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5684
I20260929 16:40:10.352797 6115 tablet_replica.cc:330] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: stopping tablet replica
I20260929 16:40:10.353434 6116 tablet_replica.cc:330] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: stopping tablet replica
I20260929 16:40:10.354115 6116 raft_consensus.cc:2229] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:40:10.354684 6115 raft_consensus.cc:2229] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:10.355562 6116 raft_consensus.cc:2258] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:10.356124 6115 raft_consensus.cc:2258] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:10.357666 6117 tablet_replica.cc:330] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: stopping tablet replica
I20260929 16:40:10.358130 6117 raft_consensus.cc:2229] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:40:10.358647 6115 ts_tablet_manager.cc:1905] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:10.358700 6117 raft_consensus.cc:2258] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:40:10.359181 6116 ts_tablet_manager.cc:1905] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:10.360946 6117 ts_tablet_manager.cc:1905] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:40:10.366209 6115 ts_tablet_manager.cc:1918] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:10.366402 6116 ts_tablet_manager.cc:1918] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:10.366456 6115 log.cc:1196] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-2/wal/wals/8fe09536d5064d0281a0d2e6db07cc20
I20260929 16:40:10.366710 6116 log.cc:1196] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-1/wal/wals/8fe09536d5064d0281a0d2e6db07cc20
I20260929 16:40:10.366948 6115 ts_tablet_manager.cc:1939] T 8fe09536d5064d0281a0d2e6db07cc20 P 7a69826257cc4cb38e3b43aa9c00e394: Deleting consensus metadata
I20260929 16:40:10.367357 6116 ts_tablet_manager.cc:1939] T 8fe09536d5064d0281a0d2e6db07cc20 P e65cfea3c1d748b5987b8d52c4f0629e: Deleting consensus metadata
I20260929 16:40:10.367806 6117 ts_tablet_manager.cc:1918] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:40:10.368001 6117 log.cc:1196] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Deleting WAL directory at /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699695913193-18228-0/minicluster-data/ts-0/wal/wals/8fe09536d5064d0281a0d2e6db07cc20
I20260929 16:40:10.368526 6117 ts_tablet_manager.cc:1939] T 8fe09536d5064d0281a0d2e6db07cc20 P 20420b836a764dfd8774e063fe37be09: Deleting consensus metadata
I20260929 16:40:10.369253 5611 catalog_manager.cc:4841] TS e65cfea3c1d748b5987b8d52c4f0629e (127.17.205.2:35449): tablet 8fe09536d5064d0281a0d2e6db07cc20 (table test-table [id=eab4e7b60be549e49c72696ba3491412]) successfully deleted
I20260929 16:40:10.369483 5612 catalog_manager.cc:4841] TS 7a69826257cc4cb38e3b43aa9c00e394 (127.17.205.3:34223): tablet 8fe09536d5064d0281a0d2e6db07cc20 (table test-table [id=eab4e7b60be549e49c72696ba3491412]) successfully deleted
I20260929 16:40:10.370168 5610 catalog_manager.cc:4841] TS 20420b836a764dfd8774e063fe37be09 (127.17.205.1:36713): tablet 8fe09536d5064d0281a0d2e6db07cc20 (table test-table [id=eab4e7b60be549e49c72696ba3491412]) successfully deleted
I20260929 16:40:10.371034 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.345219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.17.205.1:36713)
0929 16:40:10.345420 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.345427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.357967 (+ 12540us) server_negotiation.cc:407] Connection header received
0929 16:40:10.358292 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.358298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.358439 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.358537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:10.359280 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.360387 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.362552 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.363230 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.363336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.369391 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:10.369403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:10.369421 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:10.370225 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:10.370298 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:10.370325 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:10.370446 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.370597 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.370715 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:40:10.371745 5806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:10.488193 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5684
I20260929 16:40:10.502741 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5815
I20260929 16:40:10.509519 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.503506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.17.205.2:35449)
0929 16:40:10.503706 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.503712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.503732 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:40:10.503849 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.503855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.503945 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.504037 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:10.504462 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.505235 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.506488 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.506916 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.507029 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.508125 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:10.508135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:10.508150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:10.508787 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:10.508842 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:10.508864 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:10.508967 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.509093 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.509200 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:40:10.510231 5937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:10.620909 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5815
I20260929 16:40:10.636698 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5945
I20260929 16:40:10.642815 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:10.637424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.17.205.3:34223)
0929 16:40:10.637610 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:10.637614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:10.637633 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:40:10.637761 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:10.637766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:10.637868 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:10.637948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:10.638289 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.638983 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.640167 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:10.640546 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:10.640634 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:10.641536 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:10.641545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:10.641559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:10.642143 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:10.642190 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:10.642215 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:10.642300 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:10.642391 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 16:40:10.642488 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:40:10.643607 6067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:10.730307 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5945
I20260929 16:40:10.744262 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 5594
I20260929 16:40:10.745421 5655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:10.853961 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 5594
2026-09-29T16:40:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2916 ms)
[----------] 6 tests from EncryptionPolicyTest (21174 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:10 dist-test-slave-3h01 krb5kdc[6125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:10 dist-test-slave-3h01 krb5kdc[6125](info): set up 2 sockets
Sep 29 16:40:10 dist-test-slave-3h01 krb5kdc[6125](info): commencing operation
krb5kdc: starting...
W20260929 16:40:11.859716 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:12.104760 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:12.437237 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:12.437551 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:12.437691 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:12.444989 6138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:12.445120 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:12.445201 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:12.445283 6138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:12.445341 6138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:12.455541 6138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:12.456872 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:12.458262 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:12.469882 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:12.469913 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:12.470084 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:12.470847 6138 server_base.cc:1034] running on GCE node
I20260929 16:40:12.471650 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:12.473120 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:12.474356 6138 hybrid_clock.cc:648] HybridClock initialized: now 1790700012474281 us; error 51 us; skew 500 ppm
I20260929 16:40:12.474722 6138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:12.477871 6138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:12.479421 6138 webserver.cc:460] Webserver started at http://127.0.0.1:40657/ using document root <none> and password file <none>
I20260929 16:40:12.480068 6138 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:12.480170 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:12.480425 6138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:12.483481 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "352fcf534c714cee8a1aae6cac456e43"
format_stamp: "Formatted at 2026-09-29 16:40:12 on dist-test-slave-3h01"
server_key: "bc5eae0a08318aed3b3db8d25e4e796a"
server_key_iv: "cd0bd58b050261319fda30366fed2644"
server_key_version: "encryptionkey@0"
I20260929 16:40:12.484223 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "352fcf534c714cee8a1aae6cac456e43"
format_stamp: "Formatted at 2026-09-29 16:40:12 on dist-test-slave-3h01"
server_key: "bc5eae0a08318aed3b3db8d25e4e796a"
server_key_iv: "cd0bd58b050261319fda30366fed2644"
server_key_version: "encryptionkey@0"
I20260929 16:40:12.489408 6138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:12.492959 6153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:12.494306 6138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:40:12.494453 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "352fcf534c714cee8a1aae6cac456e43"
format_stamp: "Formatted at 2026-09-29 16:40:12 on dist-test-slave-3h01"
server_key: "bc5eae0a08318aed3b3db8d25e4e796a"
server_key_iv: "cd0bd58b050261319fda30366fed2644"
server_key_version: "encryptionkey@0"
I20260929 16:40:12.494625 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:12.529359 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:12.533468 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:12.533717 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:12.549947 6138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39513
I20260929 16:40:12.549973 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39513 every 8 connection(s)
I20260929 16:40:12.551366 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:12.555047 6205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:12.559386 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6138
I20260929 16:40:12.559610 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:12.560374 18228 external_mini_cluster.cc:1428] Setting key 96748420221ba0c7111792f874645340
I20260929 16:40:12.566359 6205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43: Bootstrap starting.
I20260929 16:40:12.569906 6205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:12.571000 6205 log.cc:824] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:12.574273 6205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43: No bootstrap required, opened a new log
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:40:12.582031 6205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } }
I20260929 16:40:12.582520 6205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:12.582616 6205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352fcf534c714cee8a1aae6cac456e43, State: Initialized, Role: FOLLOWER
I20260929 16:40:12.583262 6205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } }
I20260929 16:40:12.583493 6205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:12.583621 6205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:12.583726 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:12.562228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56753 (local address 127.0.0.1:39513)
0929 16:40:12.562971 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:12.562983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:12.563026 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:40:12.564838 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:12.564861 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:12.565243 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:12.565558 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:12.567250 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:12.568579 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:12.570332 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:12.571075 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:12.571281 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:12.575944 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:12.575952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:12.575971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:12.576015 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:12.579278 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:12.580063 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:12.580069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:12.580076 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:12.580251 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:12.580701 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:12.580704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:12.580707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:12.581264 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:12.581553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:12.582059 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 16:40:12.582583 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260929 16:40:12.583784 6205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:12.586030 6205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } }
I20260929 16:40:12.586571 6205 leader_election.cc:302] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352fcf534c714cee8a1aae6cac456e43; no voters:
I20260929 16:40:12.587148 6205 leader_election.cc:288] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:12.587278 6210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:12.587663 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:12.589416 6205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:12.594319 6210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [term 1 LEADER]: Becoming Leader. State: Replica: 352fcf534c714cee8a1aae6cac456e43, State: Running, Role: LEADER
I20260929 16:40:12.594945 6210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } }
I20260929 16:40:12.598994 6212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352fcf534c714cee8a1aae6cac456e43. Latest consensus state: current_term: 1 leader_uuid: "352fcf534c714cee8a1aae6cac456e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } } }
I20260929 16:40:12.599401 6212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:12.599650 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352fcf534c714cee8a1aae6cac456e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352fcf534c714cee8a1aae6cac456e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39513 } } }
I20260929 16:40:12.600349 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:12.603638 6225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:12.616400 6225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:12.619376 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:12.592414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50571 (local address 127.0.0.1:39513)
0929 16:40:12.592587 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:12.592593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:12.593078 (+ 485us) server_negotiation.cc:407] Connection header received
0929 16:40:12.593248 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:12.593253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:12.593319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:12.593395 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:12.594771 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:12.595671 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:12.603661 (+ 7990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:12.604326 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:12.604434 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:12.610558 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:12.610566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:12.610571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:12.610622 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:12.614592 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:12.615754 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:12.615761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:12.615765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:12.615930 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:12.616538 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:12.616547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:12.616551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:12.617567 (+ 1016us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:12.617866 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:12.618906 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0929 16:40:12.619053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:40:12.628051 6225 catalog_manager.cc:1347] Generated new cluster ID: 37629bb6404e455aa1e29e5d1970386b
I20260929 16:40:12.628172 6225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:12.657974 6225 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:12.659302 6225 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:12.672719 6225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 352fcf534c714cee8a1aae6cac456e43: Generated new TSK 0
I20260929 16:40:12.673871 6225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:12.689343 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6138
I20260929 16:40:12.690722 6200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:12.785338 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6138
W20260929 16:40:12.799855 6123 connection.cc:535] client connection to 127.0.0.1:39513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39513): Connection reset by peer (error 104)
2026-09-29T16:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6237](info): set up 2 sockets
Sep 29 16:40:12 dist-test-slave-3h01 krb5kdc[6237](info): commencing operation
krb5kdc: starting...
W20260929 16:40:13.799376 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:13 dist-test-slave-3h01 krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:14.036335 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:14.366358 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:14.366688 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:14.366825 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:14.374275 6250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:14.374415 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:14.374472 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:14.374567 6250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:14.374624 6250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:14.384991 6250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:14.386430 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:14.388062 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:14.400743 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:14.400695 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:14.400655 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:14.401130 6250 server_base.cc:1034] running on GCE node
I20260929 16:40:14.402218 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:14.403815 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:14.405032 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790700014404969 us; error 53 us; skew 500 ppm
I20260929 16:40:14.405378 6250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:14 dist-test-slave-3h01 krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:14.408812 6250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:14.410319 6250 webserver.cc:460] Webserver started at http://127.0.0.1:34121/ using document root <none> and password file <none>
I20260929 16:40:14.411010 6250 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:14.411145 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:14.411476 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:14.414572 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "7cb79ac801cd4854bcb6b4b6787a84b9"
format_stamp: "Formatted at 2026-09-29 16:40:14 on dist-test-slave-3h01"
server_key: "55a5a89f901aaa6d366b6728f686d776"
server_key_iv: "114d67950d4ad05f701351517d3bf479"
server_key_version: "encryptionkey@0"
I20260929 16:40:14.415390 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "7cb79ac801cd4854bcb6b4b6787a84b9"
format_stamp: "Formatted at 2026-09-29 16:40:14 on dist-test-slave-3h01"
server_key: "55a5a89f901aaa6d366b6728f686d776"
server_key_iv: "114d67950d4ad05f701351517d3bf479"
server_key_version: "encryptionkey@0"
I20260929 16:40:14.421259 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:40:14.425047 6265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:14.426613 6250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:40:14.426797 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "7cb79ac801cd4854bcb6b4b6787a84b9"
format_stamp: "Formatted at 2026-09-29 16:40:14 on dist-test-slave-3h01"
server_key: "55a5a89f901aaa6d366b6728f686d776"
server_key_iv: "114d67950d4ad05f701351517d3bf479"
server_key_version: "encryptionkey@0"
I20260929 16:40:14.426991 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:14.474962 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:14.479100 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:14.479413 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:14.496969 6250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41339
I20260929 16:40:14.496989 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41339 every 8 connection(s)
I20260929 16:40:14.498409 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:14.500260 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6250
I20260929 16:40:14.500444 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:14.501003 18228 external_mini_cluster.cc:1428] Setting key 7f8f82b5ba3080471c414d02dcacfd5c
I20260929 16:40:14.503079 6317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:14.514523 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:14.502834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45585 (local address 127.0.0.1:41339)
0929 16:40:14.503644 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:14.503656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:14.503698 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:40:14.505351 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:14.505376 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:14.505804 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:14.506156 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:14.506756 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:14.508146 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:14.509964 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:14.510679 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:14.510816 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:14.512126 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:14.512133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:14.512152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:14.512749 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:14.512841 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:14.512870 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:14.512999 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:14.513265 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:40:14.513481 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":372,"threads_started":1}
I20260929 16:40:14.516005 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9: Bootstrap starting.
I20260929 16:40:14.518890 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:14.519438 6317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:14.520468 6317 log.cc:824] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:14.523773 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9: No bootstrap required, opened a new log
I20260929 16:40:14.528916 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:14.523135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34737 (local address 127.0.0.1:41339)
0929 16:40:14.523342 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:14.523348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:14.523684 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:40:14.523871 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:14.523877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:14.523950 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:14.524030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:14.524516 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:14.525281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:14.526567 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:14.527022 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:14.527129 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:14.528235 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:14.528240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:14.528243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:14.528261 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:14.528303 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:14.528318 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:14.528418 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:14.528595 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:40:14.528651 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:40:14.531775 6317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 16:40:14.532188 6317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:14.532275 6317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb79ac801cd4854bcb6b4b6787a84b9, State: Initialized, Role: FOLLOWER
I20260929 16:40:14.532862 6317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 16:40:14.533087 6317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:14.533211 6317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:14.533365 6317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:14.535884 6317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 16:40:14.536379 6317 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb79ac801cd4854bcb6b4b6787a84b9; no voters:
I20260929 16:40:14.536916 6317 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:14.537052 6327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:14.537707 6327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb79ac801cd4854bcb6b4b6787a84b9, State: Running, Role: LEADER
I20260929 16:40:14.538208 6327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 16:40:14.538893 6317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:14.540835 6329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb79ac801cd4854bcb6b4b6787a84b9. Latest consensus state: current_term: 1 leader_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } } }
I20260929 16:40:14.541445 6329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:14.541312 6328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb79ac801cd4854bcb6b4b6787a84b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } } }
I20260929 16:40:14.542109 6328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:14.547200 6333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:14.552688 6333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:14.561610 6333 catalog_manager.cc:1347] Generated new cluster ID: 46793c7953174538a4d5d816485bebbe
I20260929 16:40:14.561731 6333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:14.574630 6333 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:14.576541 6333 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:14.584141 6333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb79ac801cd4854bcb6b4b6787a84b9: Generated new TSK 0
I20260929 16:40:14.585230 6333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:14.596221 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6250
W20260929 16:40:14.597308 6312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45585
W20260929 16:40:14.598039 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:14.598201 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6250
W20260929 16:40:14.614321 6235 connection.cc:535] client connection to 127.0.0.1:41339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41339): Connection reset by peer (error 104)
2026-09-29T16:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1813 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:14 dist-test-slave-3h01 krb5kdc[6348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:14 dist-test-slave-3h01 krb5kdc[6348](info): set up 2 sockets
Sep 29 16:40:14 dist-test-slave-3h01 krb5kdc[6348](info): commencing operation
krb5kdc: starting...
W20260929 16:40:15.611331 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:15 dist-test-slave-3h01 krb5kdc[6348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:15.857429 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:16.199261 6361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:16.199607 6361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:16.199729 6361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:16.208165 6361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:16.208308 6361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:16.208371 6361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:16.208439 6361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:16.208488 6361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:16.218808 6361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:16.220165 6361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:16.221700 6361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:16.233018 6366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:16.233062 6367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:16.233083 6369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:16.233791 6361 server_base.cc:1034] running on GCE node
I20260929 16:40:16.234540 6361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:16.235997 6361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:16.237195 6361 hybrid_clock.cc:648] HybridClock initialized: now 1790700016237148 us; error 42 us; skew 500 ppm
I20260929 16:40:16.237521 6361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:16 dist-test-slave-3h01 krb5kdc[6348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:16.245129 6361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:16.246692 6361 webserver.cc:460] Webserver started at http://127.0.0.1:43135/ using document root <none> and password file <none>
I20260929 16:40:16.247429 6361 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:16.247538 6361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:16.247792 6361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:16.250818 6361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "6f1e7e82d7584f7fb01695be506e52b9"
format_stamp: "Formatted at 2026-09-29 16:40:16 on dist-test-slave-3h01"
server_key: "29a429d381e4b32a99fea40668ceeddf"
server_key_iv: "b9a2ad6d1f25e613f129182212506eb5"
server_key_version: "encryptionkey@0"
I20260929 16:40:16.251596 6361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "6f1e7e82d7584f7fb01695be506e52b9"
format_stamp: "Formatted at 2026-09-29 16:40:16 on dist-test-slave-3h01"
server_key: "29a429d381e4b32a99fea40668ceeddf"
server_key_iv: "b9a2ad6d1f25e613f129182212506eb5"
server_key_version: "encryptionkey@0"
I20260929 16:40:16.256692 6361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:40:16.260143 6376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:16.261550 6361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:40:16.261718 6361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "6f1e7e82d7584f7fb01695be506e52b9"
format_stamp: "Formatted at 2026-09-29 16:40:16 on dist-test-slave-3h01"
server_key: "29a429d381e4b32a99fea40668ceeddf"
server_key_iv: "b9a2ad6d1f25e613f129182212506eb5"
server_key_version: "encryptionkey@0"
I20260929 16:40:16.261891 6361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:16.267645 6361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:16.271721 6361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:16.271977 6361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:16.289357 6361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33477
I20260929 16:40:16.289381 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33477 every 8 connection(s)
I20260929 16:40:16.290791 6361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:16.294516 6428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:16.300876 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6361
I20260929 16:40:16.301061 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:16.301591 18228 external_mini_cluster.cc:1428] Setting key 038e03f9abce9900b3d48e2c42e4c7f5
I20260929 16:40:16.305729 6428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9: Bootstrap starting.
I20260929 16:40:16.308976 6428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:16.309302 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:16.303475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37731 (local address 127.0.0.1:33477)
0929 16:40:16.304258 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:16.304271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:16.304313 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:40:16.305996 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:16.306019 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:16.306386 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:16.306746 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:16.307738 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:16.307747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:16.307761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:16.307775 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:16.307846 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:16.307871 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:16.308063 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:16.308185 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:40:16.308366 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260929 16:40:16.310050 6428 log.cc:824] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:16.312896 6428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9: No bootstrap required, opened a new log
I20260929 16:40:16.312892 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:16.319320 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:16.316911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45883 (local address 127.0.0.1:33477)
0929 16:40:16.317054 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:16.317059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:16.317489 (+ 430us) server_negotiation.cc:407] Connection header received
0929 16:40:16.317653 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:16.317657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:16.317713 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:16.317800 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:16.318664 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:16.318677 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:16.318682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:16.318696 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:16.318733 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:16.318748 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:16.318860 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:16.318966 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:40:16.319028 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:40:16.320981 6428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260929 16:40:16.321331 6428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:16.321420 6428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1e7e82d7584f7fb01695be506e52b9, State: Initialized, Role: FOLLOWER
I20260929 16:40:16.322006 6428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260929 16:40:16.322280 6428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:16.322407 6428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:16.322598 6428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:16.324769 6428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260929 16:40:16.325222 6428 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1e7e82d7584f7fb01695be506e52b9; no voters:
I20260929 16:40:16.325758 6428 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:16.325898 6438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:16.326565 6438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1e7e82d7584f7fb01695be506e52b9, State: Running, Role: LEADER
I20260929 16:40:16.327150 6438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260929 16:40:16.327756 6428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:16.329854 6440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1e7e82d7584f7fb01695be506e52b9. Latest consensus state: current_term: 1 leader_uuid: "6f1e7e82d7584f7fb01695be506e52b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } } }
I20260929 16:40:16.330188 6440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:16.330978 6443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:16.330865 6439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1e7e82d7584f7fb01695be506e52b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e7e82d7584f7fb01695be506e52b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } } }
I20260929 16:40:16.335304 6439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:16.337370 6443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:16.346356 6443 catalog_manager.cc:1347] Generated new cluster ID: bdc6c2de77ba42f09c3cf255c1fcf804
I20260929 16:40:16.346473 6443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:16.364459 6443 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:16.365020 6443 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:16.377661 6443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1e7e82d7584f7fb01695be506e52b9: Generated new TSK 0
I20260929 16:40:16.378539 6443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:16.385483 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6361
W20260929 16:40:16.387128 6423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37731
W20260929 16:40:16.387904 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6361: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:16.388090 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6361
2026-09-29T16:40:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:16 dist-test-slave-3h01 krb5kdc[6459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:16 dist-test-slave-3h01 krb5kdc[6459](info): set up 2 sockets
Sep 29 16:40:16 dist-test-slave-3h01 krb5kdc[6459](info): commencing operation
krb5kdc: starting...
W20260929 16:40:17.395437 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:17 dist-test-slave-3h01 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:17.632104 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:17.966609 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:17.966917 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:17.967044 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:17.974538 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:17.974668 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:17.974733 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:17.974818 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:17.974876 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:17.985105 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:17.986487 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:17.987941 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:17.999586 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:17.999693 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:17.999729 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:18.000483 6472 server_base.cc:1034] running on GCE node
I20260929 16:40:18.001236 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:18.002681 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:18.003860 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790700018003810 us; error 44 us; skew 500 ppm
I20260929 16:40:18.004181 6472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:18 dist-test-slave-3h01 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:18.007496 6472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:18.009099 6472 webserver.cc:460] Webserver started at http://127.0.0.1:43305/ using document root <none> and password file <none>
I20260929 16:40:18.009847 6472 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:18.009968 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:18.010277 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:18.013370 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "5b0993dc74e147d2bd0452357e0d7530"
format_stamp: "Formatted at 2026-09-29 16:40:18 on dist-test-slave-3h01"
server_key: "f76a3283e559a540cef406e658053c7d"
server_key_iv: "2ac116679c630d60d17e4c88025fec7b"
server_key_version: "encryptionkey@0"
I20260929 16:40:18.014174 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "5b0993dc74e147d2bd0452357e0d7530"
format_stamp: "Formatted at 2026-09-29 16:40:18 on dist-test-slave-3h01"
server_key: "f76a3283e559a540cef406e658053c7d"
server_key_iv: "2ac116679c630d60d17e4c88025fec7b"
server_key_version: "encryptionkey@0"
I20260929 16:40:18.019503 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:40:18.023468 6487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:18.024992 6472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:18.025197 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "5b0993dc74e147d2bd0452357e0d7530"
format_stamp: "Formatted at 2026-09-29 16:40:18 on dist-test-slave-3h01"
server_key: "f76a3283e559a540cef406e658053c7d"
server_key_iv: "2ac116679c630d60d17e4c88025fec7b"
server_key_version: "encryptionkey@0"
I20260929 16:40:18.025403 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:18.051422 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:18.055891 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:18.056171 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:18.073623 6472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39937
I20260929 16:40:18.073654 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39937 every 8 connection(s)
I20260929 16:40:18.075076 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:18.079020 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:18.085386 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6472
I20260929 16:40:18.085558 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:18.085999 18228 external_mini_cluster.cc:1428] Setting key dd4018a9cf738f6ae4de2ccc722f1657
I20260929 16:40:18.091125 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530: Bootstrap starting.
I20260929 16:40:18.095007 6539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:18.096252 6539 log.cc:824] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:18.099524 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530: No bootstrap required, opened a new log
Sep 29 16:40:18 dist-test-slave-3h01 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:40:18.109416 6539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260929 16:40:18.109886 6539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:18.110002 6539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0993dc74e147d2bd0452357e0d7530, State: Initialized, Role: FOLLOWER
I20260929 16:40:18.110723 6539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260929 16:40:18.110917 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:18.088009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45329 (local address 127.0.0.1:39937)
0929 16:40:18.088887 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:18.088904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:18.088955 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:40:18.091304 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:18.091336 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:18.091899 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:18.092350 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:18.093709 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:18.095708 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:18.097107 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:18.097992 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:18.098187 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:18.102010 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:18.102023 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:18.102045 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:18.102102 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:18.106105 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:18.106826 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:18.106836 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:18.106853 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:18.107089 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:18.107467 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:18.107472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:18.107476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:18.108273 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:18.108564 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:18.108990 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 16:40:18.109450 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":367,"threads_started":1}
I20260929 16:40:18.111016 6539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:18.111153 6539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:18.111361 6539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:18.114235 6539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260929 16:40:18.114852 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:18.114787 6539 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0993dc74e147d2bd0452357e0d7530; no voters:
I20260929 16:40:18.115459 6539 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:18.115581 6544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:18.117362 6539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:18.124140 6544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0993dc74e147d2bd0452357e0d7530, State: Running, Role: LEADER
I20260929 16:40:18.125015 6544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260929 16:40:18.128762 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0993dc74e147d2bd0452357e0d7530. Latest consensus state: current_term: 1 leader_uuid: "5b0993dc74e147d2bd0452357e0d7530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } } }
I20260929 16:40:18.129161 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:18.129956 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0993dc74e147d2bd0452357e0d7530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0993dc74e147d2bd0452357e0d7530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } } }
I20260929 16:40:18.130260 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:18.132287 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:18.137737 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:18.140110 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:18.125607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50187 (local address 127.0.0.1:39937)
0929 16:40:18.125768 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:18.125774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:18.126529 (+ 755us) server_negotiation.cc:407] Connection header received
0929 16:40:18.126797 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:18.126801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:18.126865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:18.126946 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:18.128762 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:18.129701 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:18.131198 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:18.131965 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:18.132091 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:18.134604 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:18.134610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:18.134613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:18.134647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:18.137419 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:18.138134 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:18.138140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:18.138142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:18.138261 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:18.138614 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:18.138619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:18.138622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:18.138940 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:18.139149 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:18.139544 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:40:18.139759 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:40:18.147169 6562 catalog_manager.cc:1347] Generated new cluster ID: e66cb79a6884469c911b42b2a3c82408
I20260929 16:40:18.147295 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:18.158064 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:18.160007 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:18.171144 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0993dc74e147d2bd0452357e0d7530: Generated new TSK 0
I20260929 16:40:18.172314 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:18.186527 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6472
I20260929 16:40:18.188045 6534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:18.279012 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6472
2026-09-29T16:40:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:18 dist-test-slave-3h01 krb5kdc[6571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:18 dist-test-slave-3h01 krb5kdc[6571](info): set up 2 sockets
Sep 29 16:40:18 dist-test-slave-3h01 krb5kdc[6571](info): commencing operation
krb5kdc: starting...
W20260929 16:40:19.291615 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:19 dist-test-slave-3h01 krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:19.534392 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:19.856604 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:19.856930 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:19.857074 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:19.864347 6584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:19.864485 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:19.864562 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:19.864642 6584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:19.864701 6584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:19.874670 6584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:19.876022 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:19.877336 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:19.888705 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:19.888680 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:19.888741 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:19.889395 6584 server_base.cc:1034] running on GCE node
I20260929 16:40:19.890184 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:19.891677 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:19.892884 6584 hybrid_clock.cc:648] HybridClock initialized: now 1790700019892829 us; error 50 us; skew 500 ppm
I20260929 16:40:19.893214 6584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:19 dist-test-slave-3h01 krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:19.896317 6584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:19.897898 6584 webserver.cc:460] Webserver started at http://127.0.0.1:42749/ using document root <none> and password file <none>
I20260929 16:40:19.898608 6584 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:19.898736 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:19.899027 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:19.902097 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "8e899d9816804e1fa955d5e20da5fae4"
format_stamp: "Formatted at 2026-09-29 16:40:19 on dist-test-slave-3h01"
server_key: "ae4b1f7e3135c1b12dbab800bb96aad6"
server_key_iv: "8949bb12fb5a296848ec7e6267dd9ab7"
server_key_version: "encryptionkey@0"
I20260929 16:40:19.902889 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "8e899d9816804e1fa955d5e20da5fae4"
format_stamp: "Formatted at 2026-09-29 16:40:19 on dist-test-slave-3h01"
server_key: "ae4b1f7e3135c1b12dbab800bb96aad6"
server_key_iv: "8949bb12fb5a296848ec7e6267dd9ab7"
server_key_version: "encryptionkey@0"
I20260929 16:40:19.908270 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:40:19.911962 6599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:19.913426 6584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:19.913610 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "8e899d9816804e1fa955d5e20da5fae4"
format_stamp: "Formatted at 2026-09-29 16:40:19 on dist-test-slave-3h01"
server_key: "ae4b1f7e3135c1b12dbab800bb96aad6"
server_key_iv: "8949bb12fb5a296848ec7e6267dd9ab7"
server_key_version: "encryptionkey@0"
I20260929 16:40:19.913810 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:19.944643 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:19.948897 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:19.949174 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:19.966446 6584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33353
I20260929 16:40:19.966468 6650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33353 every 8 connection(s)
I20260929 16:40:19.967942 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:19.971767 6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:19.976234 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6584
I20260929 16:40:19.976424 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:19.976940 18228 external_mini_cluster.cc:1428] Setting key 846135541b1feb9b0790922a91bc80fc
I20260929 16:40:19.983376 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4: Bootstrap starting.
I20260929 16:40:19.986946 6651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:19.988116 6651 log.cc:824] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:19.990824 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:19.978847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39863 (local address 127.0.0.1:33353)
0929 16:40:19.979675 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:19.979688 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:19.979732 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:40:19.981786 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:19.981812 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:19.982249 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:19.982604 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:19.983180 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:19.984558 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:19.986367 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:19.987080 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:19.987286 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:19.988391 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:19.988399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:19.988414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:19.989113 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:19.989211 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:19.989249 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:19.989386 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:19.989582 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 16:40:19.989863 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":381,"threads_started":1}
I20260929 16:40:19.991240 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4: No bootstrap required, opened a new log
I20260929 16:40:19.994443 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:20.000900 6651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } }
I20260929 16:40:20.001343 6651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:20.001447 6651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e899d9816804e1fa955d5e20da5fae4, State: Initialized, Role: FOLLOWER
I20260929 16:40:20.002107 6651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } }
I20260929 16:40:20.002401 6651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:20.002558 6651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:20.002740 6651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:20.005762 6651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } }
I20260929 16:40:20.005932 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:19.999489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45545 (local address 127.0.0.1:33353)
0929 16:40:19.999700 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:19.999707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:20.000102 (+ 395us) server_negotiation.cc:407] Connection header received
0929 16:40:20.000257 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:20.000261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:20.000322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:20.000395 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:20.001037 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:20.001841 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:20.003623 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:20.004117 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:20.004215 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:20.005273 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:20.005279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:20.005282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:20.005300 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:20.005338 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:20.005355 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:20.005419 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:20.005543 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:40:20.005614 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:40:20.006330 6651 leader_election.cc:302] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e899d9816804e1fa955d5e20da5fae4; no voters:
I20260929 16:40:20.006970 6651 leader_election.cc:288] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:20.007083 6661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:20.007689 6661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e899d9816804e1fa955d5e20da5fae4, State: Running, Role: LEADER
I20260929 16:40:20.008292 6661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } }
I20260929 16:40:20.009368 6651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:20.016228 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e899d9816804e1fa955d5e20da5fae4. Latest consensus state: current_term: 1 leader_uuid: "8e899d9816804e1fa955d5e20da5fae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } } }
I20260929 16:40:20.016466 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e899d9816804e1fa955d5e20da5fae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e899d9816804e1fa955d5e20da5fae4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33353 } } }
I20260929 16:40:20.016718 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:20.016737 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:20.018546 6673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:20.022718 6673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:20.031919 6673 catalog_manager.cc:1347] Generated new cluster ID: d5f6dc4d3e1d4d7b9fd5134dd1413167
I20260929 16:40:20.032055 6673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:20.043049 6673 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:20.044997 6673 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:20.059233 6673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e899d9816804e1fa955d5e20da5fae4: Generated new TSK 0
I20260929 16:40:20.060098 6673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:20.069186 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6584
W20260929 16:40:20.070384 6646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39863
W20260929 16:40:20.071022 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:20.071148 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6584
2026-09-29T16:40:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1792 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:20 dist-test-slave-3h01 krb5kdc[6682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:20 dist-test-slave-3h01 krb5kdc[6682](info): set up 2 sockets
Sep 29 16:40:20 dist-test-slave-3h01 krb5kdc[6682](info): commencing operation
krb5kdc: starting...
W20260929 16:40:21.083406 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:21 dist-test-slave-3h01 krb5kdc[6682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:21.329665 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:21.665702 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:21.666109 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:21.666272 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:21.676017 6695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:21.676230 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:21.676321 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:21.676410 6695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:21.676492 6695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:21.688165 6695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:21.689478 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:21.690829 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:21.702732 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:21.702762 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:21.702864 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:21.703392 6695 server_base.cc:1034] running on GCE node
I20260929 16:40:21.704137 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:21.705535 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:21.706751 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790700021706701 us; error 41 us; skew 500 ppm
I20260929 16:40:21.707084 6695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:21 dist-test-slave-3h01 krb5kdc[6682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:21.713248 6695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:21.714782 6695 webserver.cc:460] Webserver started at http://127.0.0.1:41879/ using document root <none> and password file <none>
I20260929 16:40:21.715531 6695 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:21.715634 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:21.715903 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:21.718847 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "41ba1fdf352d4a39887555007bf0652f"
format_stamp: "Formatted at 2026-09-29 16:40:21 on dist-test-slave-3h01"
server_key: "c729bd63b23f8de7a1e23731d389016d"
server_key_iv: "0faff25e084302c1b1b718824260122a"
server_key_version: "encryptionkey@0"
I20260929 16:40:21.719627 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "41ba1fdf352d4a39887555007bf0652f"
format_stamp: "Formatted at 2026-09-29 16:40:21 on dist-test-slave-3h01"
server_key: "c729bd63b23f8de7a1e23731d389016d"
server_key_iv: "0faff25e084302c1b1b718824260122a"
server_key_version: "encryptionkey@0"
I20260929 16:40:21.724707 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:40:21.728492 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:21.729887 6695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:21.730037 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "41ba1fdf352d4a39887555007bf0652f"
format_stamp: "Formatted at 2026-09-29 16:40:21 on dist-test-slave-3h01"
server_key: "c729bd63b23f8de7a1e23731d389016d"
server_key_iv: "0faff25e084302c1b1b718824260122a"
server_key_version: "encryptionkey@0"
I20260929 16:40:21.730211 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:21.736078 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:21.739965 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:21.740211 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:21.757218 6695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39691
I20260929 16:40:21.757249 6761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39691 every 8 connection(s)
I20260929 16:40:21.758695 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:21.762212 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6695
I20260929 16:40:21.762403 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:21.762794 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:21.762943 18228 external_mini_cluster.cc:1428] Setting key ed0397499815a7cd8bc81d1bf9a32b47
I20260929 16:40:21.770828 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:21.764743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57085 (local address 127.0.0.1:39691)
0929 16:40:21.765503 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:21.765516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:21.765559 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:40:21.767557 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:21.767582 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:21.767948 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:21.768289 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:21.769293 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:21.769299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:21.769311 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:21.769324 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:21.769391 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:21.769410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:21.769574 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:21.769677 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:40:21.769899 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260929 16:40:21.774501 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:21.776319 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f: Bootstrap starting.
I20260929 16:40:21.780254 6762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:21.781558 6762 log.cc:824] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:21.781838 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:21.779538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47179 (local address 127.0.0.1:39691)
0929 16:40:21.779681 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:21.779685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:21.780031 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:40:21.780200 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:21.780203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:21.780253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:21.780333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:21.781216 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:21.781220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:21.781226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:21.781240 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:21.781276 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:21.781290 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:21.781373 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:21.781486 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 16:40:21.781528 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:40:21.784478 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f: No bootstrap required, opened a new log
I20260929 16:40:21.791582 6762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260929 16:40:21.791960 6762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:21.792055 6762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ba1fdf352d4a39887555007bf0652f, State: Initialized, Role: FOLLOWER
I20260929 16:40:21.792615 6762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260929 16:40:21.792819 6762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:21.792968 6762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:21.793145 6762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:21.795331 6762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260929 16:40:21.795804 6762 leader_election.cc:302] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41ba1fdf352d4a39887555007bf0652f; no voters:
I20260929 16:40:21.796346 6762 leader_election.cc:288] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:21.796454 6772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:21.797055 6772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [term 1 LEADER]: Becoming Leader. State: Replica: 41ba1fdf352d4a39887555007bf0652f, State: Running, Role: LEADER
I20260929 16:40:21.797577 6772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260929 16:40:21.798287 6762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:21.800246 6774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41ba1fdf352d4a39887555007bf0652f. Latest consensus state: current_term: 1 leader_uuid: "41ba1fdf352d4a39887555007bf0652f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260929 16:40:21.800681 6774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:21.801403 6773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41ba1fdf352d4a39887555007bf0652f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba1fdf352d4a39887555007bf0652f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260929 16:40:21.801697 6773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:21.803160 6781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:21.807550 6781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:21.815822 6781 catalog_manager.cc:1347] Generated new cluster ID: c9f8ad66190447db949ebb00f35434af
I20260929 16:40:21.815939 6781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:21.828013 6781 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:21.828411 6781 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:21.838411 6781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41ba1fdf352d4a39887555007bf0652f: Generated new TSK 0
I20260929 16:40:21.839246 6781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:21.851792 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6695
W20260929 16:40:21.852900 6757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57085
W20260929 16:40:21.853667 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:21.853824 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6695
2026-09-29T16:40:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1782 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:21 dist-test-slave-3h01 krb5kdc[6793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:21 dist-test-slave-3h01 krb5kdc[6793](info): set up 2 sockets
Sep 29 16:40:21 dist-test-slave-3h01 krb5kdc[6793](info): commencing operation
krb5kdc: starting...
W20260929 16:40:22.863497 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:23.120394 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:23.450696 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:23.451035 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:23.451184 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:23.458573 6806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:23.458753 6806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:23.458837 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:23.458920 6806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:23.458979 6806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:23.469226 6806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:23.470579 6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:23.471992 6806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:23.483705 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:23.483779 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:23.483915 6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:23.484822 6806 server_base.cc:1034] running on GCE node
I20260929 16:40:23.485646 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:23.487138 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:23.488382 6806 hybrid_clock.cc:648] HybridClock initialized: now 1790700023488325 us; error 41 us; skew 500 ppm
I20260929 16:40:23.488747 6806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:23.491884 6806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:23.493358 6806 webserver.cc:460] Webserver started at http://127.0.0.1:45921/ using document root <none> and password file <none>
I20260929 16:40:23.494032 6806 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:23.494132 6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:23.494405 6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:23.497467 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "e6d12c3fe3364d558655996227061473"
format_stamp: "Formatted at 2026-09-29 16:40:23 on dist-test-slave-3h01"
server_key: "98b101be23edf8e5918d1e41a7223197"
server_key_iv: "99d6b571136bd36b1119cbb67056add4"
server_key_version: "encryptionkey@0"
I20260929 16:40:23.498198 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "e6d12c3fe3364d558655996227061473"
format_stamp: "Formatted at 2026-09-29 16:40:23 on dist-test-slave-3h01"
server_key: "98b101be23edf8e5918d1e41a7223197"
server_key_iv: "99d6b571136bd36b1119cbb67056add4"
server_key_version: "encryptionkey@0"
I20260929 16:40:23.503587 6806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:40:23.507314 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:23.508795 6806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:23.508971 6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "e6d12c3fe3364d558655996227061473"
format_stamp: "Formatted at 2026-09-29 16:40:23 on dist-test-slave-3h01"
server_key: "98b101be23edf8e5918d1e41a7223197"
server_key_iv: "99d6b571136bd36b1119cbb67056add4"
server_key_version: "encryptionkey@0"
I20260929 16:40:23.509166 6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:23.539321 6806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:23.543793 6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:23.544075 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:23.561499 6806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46407
I20260929 16:40:23.561522 6872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46407 every 8 connection(s)
I20260929 16:40:23.562942 6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:23.566740 6873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:23.573073 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6806
I20260929 16:40:23.573236 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:23.573649 18228 external_mini_cluster.cc:1428] Setting key b29b2b9409c7d2cfbba7346b8d081bbd
I20260929 16:40:23.577854 6873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473: Bootstrap starting.
I20260929 16:40:23.581745 6873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:23.583037 6873 log.cc:824] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:23.586585 6873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473: No bootstrap required, opened a new log
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:40:23.594779 6873 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } }
I20260929 16:40:23.595242 6873 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:23.595366 6873 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d12c3fe3364d558655996227061473, State: Initialized, Role: FOLLOWER
I20260929 16:40:23.596123 6873 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } }
I20260929 16:40:23.596410 6873 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:23.596558 6873 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:23.596726 6873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:23.597566 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:23.575538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:33939 (local address 127.0.0.1:46407)
0929 16:40:23.576337 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:23.576357 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:23.576400 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:40:23.578108 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:23.578133 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:23.578641 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:23.578998 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:23.580955 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:23.582328 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:23.584145 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:23.585007 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:23.585195 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:23.589888 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:23.589896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:23.589918 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:23.589965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:23.593130 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:23.593980 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:23.593988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:23.593995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:23.594176 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:23.594637 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:23.594640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:23.594643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:23.595200 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:23.595566 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:23.596020 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:40:23.596360 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260929 16:40:23.599120 6873 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } }
I20260929 16:40:23.599663 6873 leader_election.cc:302] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6d12c3fe3364d558655996227061473; no voters:
I20260929 16:40:23.600354 6873 leader_election.cc:288] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:23.600445 6878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:23.601619 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:23.601675 6878 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [term 1 LEADER]: Becoming Leader. State: Replica: e6d12c3fe3364d558655996227061473, State: Running, Role: LEADER
I20260929 16:40:23.602280 6878 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } }
I20260929 16:40:23.602725 6873 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:23.605175 6880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d12c3fe3364d558655996227061473. Latest consensus state: current_term: 1 leader_uuid: "e6d12c3fe3364d558655996227061473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } } }
I20260929 16:40:23.605624 6880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:23.614641 6884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:23.614959 6879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6d12c3fe3364d558655996227061473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d12c3fe3364d558655996227061473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46407 } } }
I20260929 16:40:23.615428 6879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:23.624670 6884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:23.637486 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:23.616778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:49663 (local address 127.0.0.1:46407)
0929 16:40:23.616981 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:23.616988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:23.620118 (+ 3130us) server_negotiation.cc:407] Connection header received
0929 16:40:23.620298 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:23.620304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:23.620402 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:23.620523 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:23.622635 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:23.623868 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:23.625788 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:23.626484 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:23.626589 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:23.629936 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:23.629943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:23.629948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:23.629995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:23.633649 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:23.634722 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:23.634728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:23.634731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:23.634901 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:23.635510 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:23.635515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:23.635518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:23.635964 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:23.636218 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:23.636913 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 16:40:23.637111 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:40:23.642482 6901 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:40:23.644214 6884 catalog_manager.cc:1347] Generated new cluster ID: d4f1e80710294d108b1c9f28d51edcd5
I20260929 16:40:23.644323 6884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:23.667281 6884 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:23.669201 6884 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:23.677141 6884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6d12c3fe3364d558655996227061473: Generated new TSK 0
I20260929 16:40:23.678001 6884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:23.685781 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6806
I20260929 16:40:23.686929 6868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:23.779878 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6806
W20260929 16:40:23.794286 6791 connection.cc:535] client connection to 127.0.0.1:46407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46407): Connection reset by peer (error 104)
2026-09-29T16:40:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6905](info): set up 2 sockets
Sep 29 16:40:23 dist-test-slave-3h01 krb5kdc[6905](info): commencing operation
krb5kdc: starting...
W20260929 16:40:24.791354 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:24 dist-test-slave-3h01 krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:25.030874 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:25.361270 6918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:25.361619 6918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:25.361759 6918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:25.369297 6918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:25.369442 6918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:25.369525 6918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:25.369611 6918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:25.369671 6918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:25.379959 6918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:25.381374 6918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:25.382892 6918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:25.394680 6924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:25.394876 6926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:25.395190 6923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:25.395493 6918 server_base.cc:1034] running on GCE node
I20260929 16:40:25.396332 6918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:25.397846 6918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:25.399076 6918 hybrid_clock.cc:648] HybridClock initialized: now 1790700025399017 us; error 45 us; skew 500 ppm
I20260929 16:40:25.399492 6918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:25 dist-test-slave-3h01 krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:25.402824 6918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:25.404507 6918 webserver.cc:460] Webserver started at http://127.0.0.1:32913/ using document root <none> and password file <none>
I20260929 16:40:25.405241 6918 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:25.405372 6918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:25.405715 6918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:25.408989 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1"
format_stamp: "Formatted at 2026-09-29 16:40:25 on dist-test-slave-3h01"
server_key: "12500f7f08a326bb46d15817bca61c7d"
server_key_iv: "039285e85257899c7659888e6f5e93f1"
server_key_version: "encryptionkey@0"
I20260929 16:40:25.409807 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1"
format_stamp: "Formatted at 2026-09-29 16:40:25 on dist-test-slave-3h01"
server_key: "12500f7f08a326bb46d15817bca61c7d"
server_key_iv: "039285e85257899c7659888e6f5e93f1"
server_key_version: "encryptionkey@0"
I20260929 16:40:25.415453 6918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:40:25.418908 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:25.420254 6918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:25.420413 6918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1"
format_stamp: "Formatted at 2026-09-29 16:40:25 on dist-test-slave-3h01"
server_key: "12500f7f08a326bb46d15817bca61c7d"
server_key_iv: "039285e85257899c7659888e6f5e93f1"
server_key_version: "encryptionkey@0"
I20260929 16:40:25.420593 6918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:25.446843 6918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:25.451067 6918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:25.451359 6918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:25.468087 6918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43519
I20260929 16:40:25.468118 6984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43519 every 8 connection(s)
I20260929 16:40:25.469413 6918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:25.473121 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:25.474318 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 6918
I20260929 16:40:25.474470 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:25.474922 18228 external_mini_cluster.cc:1428] Setting key 387a255522890c916cfb723d968c3657
I20260929 16:40:25.485410 6985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: Bootstrap starting.
I20260929 16:40:25.487582 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:25.476794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:59929 (local address 127.0.0.1:43519)
0929 16:40:25.477576 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:25.477588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:25.477631 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:40:25.479283 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:25.479312 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:25.479767 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:25.480186 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:25.480700 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:25.482008 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:25.483307 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:25.483984 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:25.484151 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:25.485328 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:25.485335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:25.485350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:25.485907 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:25.485980 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:25.486008 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:25.486147 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:25.486424 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:40:25.486575 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":330,"threads_started":1}
I20260929 16:40:25.488890 6985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:25.489969 6985 log.cc:824] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:25.491395 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:25.492736 6985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: No bootstrap required, opened a new log
I20260929 16:40:25.501590 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:25.495447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:51055 (local address 127.0.0.1:43519)
0929 16:40:25.495626 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:25.495632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:25.495934 (+ 302us) server_negotiation.cc:407] Connection header received
0929 16:40:25.496067 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:25.496073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:25.496155 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:25.496269 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:25.496682 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:25.497817 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:25.499033 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:25.499732 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:25.499865 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:25.500853 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:25.500858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:25.500863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:25.500884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:25.500945 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:25.500965 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:25.501069 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:25.501237 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:40:25.501289 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:40:25.501627 6985 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260929 16:40:25.502115 6985 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:25.502224 6985 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30dc1a3af74439b8ebf5cfbd0ff0ca1, State: Initialized, Role: FOLLOWER
I20260929 16:40:25.502839 6985 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260929 16:40:25.503077 6985 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:25.503201 6985 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:25.503409 6985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:25.505502 6985 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260929 16:40:25.505990 6985 leader_election.cc:302] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30dc1a3af74439b8ebf5cfbd0ff0ca1; no voters:
I20260929 16:40:25.506579 6985 leader_election.cc:288] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:25.506670 6995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:25.508545 6985 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:25.513705 6995 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [term 1 LEADER]: Becoming Leader. State: Replica: c30dc1a3af74439b8ebf5cfbd0ff0ca1, State: Running, Role: LEADER
I20260929 16:40:25.514235 6995 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260929 16:40:25.517261 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30dc1a3af74439b8ebf5cfbd0ff0ca1. Latest consensus state: current_term: 1 leader_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260929 16:40:25.517453 6996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30dc1a3af74439b8ebf5cfbd0ff0ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260929 16:40:25.517807 6996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:25.517815 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1 [sys.catalog]: This master's current role is: LEADER
W20260929 16:40:25.523666 7009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:25.523801 7009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:25.523916 7010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:25.525590 7010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:25.534144 7010 catalog_manager.cc:1347] Generated new cluster ID: e08902788dd1440290e9824b376903c9
I20260929 16:40:25.534271 7010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:25.552028 7010 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:25.553301 7010 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:25.563838 7010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30dc1a3af74439b8ebf5cfbd0ff0ca1: Generated new TSK 0
I20260929 16:40:25.564924 7010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:25.581669 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6918
W20260929 16:40:25.582820 6980 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:59929
W20260929 16:40:25.583549 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 6918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:25.583714 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 6918
W20260929 16:40:25.597903 6903 connection.cc:535] client connection to 127.0.0.1:43519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43519): Connection reset by peer (error 104)
2026-09-29T16:40:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1803 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:25 dist-test-slave-3h01 krb5kdc[7016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:25 dist-test-slave-3h01 krb5kdc[7016](info): set up 2 sockets
Sep 29 16:40:25 dist-test-slave-3h01 krb5kdc[7016](info): commencing operation
krb5kdc: starting...
W20260929 16:40:26.591435 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:26 dist-test-slave-3h01 krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:26.832415 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:27.162384 7029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:27.162739 7029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:27.162879 7029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:27.170271 7029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:27.170425 7029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:27.170506 7029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:27.170585 7029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:27.170643 7029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:27.180989 7029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:27.182389 7029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:27.183874 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:27.195808 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:27.195856 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:27.195876 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:27.196796 7029 server_base.cc:1034] running on GCE node
I20260929 16:40:27.197641 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:27.199144 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:27.200387 7029 hybrid_clock.cc:648] HybridClock initialized: now 1790700027200343 us; error 49 us; skew 500 ppm
I20260929 16:40:27.200745 7029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:27 dist-test-slave-3h01 krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:27.216327 7029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:27.218837 7029 webserver.cc:460] Webserver started at http://127.0.0.1:34821/ using document root <none> and password file <none>
I20260929 16:40:27.219638 7029 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:27.219770 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:27.220088 7029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:27.223248 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "546d8bef9d06488d889eab1c166033b9"
format_stamp: "Formatted at 2026-09-29 16:40:27 on dist-test-slave-3h01"
server_key: "3d56d3344a959b8719bd7088d7820461"
server_key_iv: "97e004793eaa66f5d41c429f6961cea0"
server_key_version: "encryptionkey@0"
I20260929 16:40:27.224071 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "546d8bef9d06488d889eab1c166033b9"
format_stamp: "Formatted at 2026-09-29 16:40:27 on dist-test-slave-3h01"
server_key: "3d56d3344a959b8719bd7088d7820461"
server_key_iv: "97e004793eaa66f5d41c429f6961cea0"
server_key_version: "encryptionkey@0"
I20260929 16:40:27.229482 7029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:27.233162 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:27.234588 7029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:40:27.234768 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "546d8bef9d06488d889eab1c166033b9"
format_stamp: "Formatted at 2026-09-29 16:40:27 on dist-test-slave-3h01"
server_key: "3d56d3344a959b8719bd7088d7820461"
server_key_iv: "97e004793eaa66f5d41c429f6961cea0"
server_key_version: "encryptionkey@0"
I20260929 16:40:27.234965 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:27.240765 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:27.245288 7029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:27.245628 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:27.263039 7029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36839
I20260929 16:40:27.263070 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36839 every 8 connection(s)
I20260929 16:40:27.264568 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:27.265938 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7029
I20260929 16:40:27.266125 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:27.266664 18228 external_mini_cluster.cc:1428] Setting key 177cf91e60bfb1ad33975aa2fda82e4b
I20260929 16:40:27.269474 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:27.275779 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:27.268621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:39855 (local address 127.0.0.1:36839)
0929 16:40:27.269436 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:27.269453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:27.269512 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:40:27.271755 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:27.271786 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:27.272245 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:27.272734 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:27.273728 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:27.273740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:27.273758 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:27.273779 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:27.273864 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:27.273891 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:27.274113 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:27.274239 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:40:27.274468 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":327,"threads_started":1}
I20260929 16:40:27.279458 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:27.283491 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9: Bootstrap starting.
I20260929 16:40:27.287228 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:27.284655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:47819 (local address 127.0.0.1:36839)
0929 16:40:27.284833 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:27.284839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:27.285368 (+ 529us) server_negotiation.cc:407] Connection header received
0929 16:40:27.285537 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:27.285543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:27.285609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:27.285730 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:27.286574 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:27.286580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:27.286585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:27.286605 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:27.286658 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:27.286679 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:27.286792 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:27.286865 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:40:27.286927 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:40:27.287434 7096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:27.288552 7096 log.cc:824] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:27.291472 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9: No bootstrap required, opened a new log
I20260929 16:40:27.299113 7096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260929 16:40:27.299553 7096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:27.299633 7096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546d8bef9d06488d889eab1c166033b9, State: Initialized, Role: FOLLOWER
I20260929 16:40:27.300221 7096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260929 16:40:27.300505 7096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:27.300652 7096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:27.300820 7096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:27.303081 7096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260929 16:40:27.303619 7096 leader_election.cc:302] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 546d8bef9d06488d889eab1c166033b9; no voters:
I20260929 16:40:27.304180 7096 leader_election.cc:288] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:27.304311 7106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:27.304898 7106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [term 1 LEADER]: Becoming Leader. State: Replica: 546d8bef9d06488d889eab1c166033b9, State: Running, Role: LEADER
I20260929 16:40:27.305418 7106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260929 16:40:27.306166 7096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:27.309783 7107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "546d8bef9d06488d889eab1c166033b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } } }
I20260929 16:40:27.310237 7107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:27.315019 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 546d8bef9d06488d889eab1c166033b9. Latest consensus state: current_term: 1 leader_uuid: "546d8bef9d06488d889eab1c166033b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546d8bef9d06488d889eab1c166033b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } } }
I20260929 16:40:27.315546 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:27.316011 7117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:27.321417 7117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:27.330613 7117 catalog_manager.cc:1347] Generated new cluster ID: 12a5dcb3765c430bab79f15695fc2427
I20260929 16:40:27.330744 7117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:27.346668 7117 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:27.347128 7117 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:27.356597 7117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 546d8bef9d06488d889eab1c166033b9: Generated new TSK 0
I20260929 16:40:27.357482 7117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:27.369007 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7029
W20260929 16:40:27.370280 7091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:39855
W20260929 16:40:27.371059 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:27.371249 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7029
2026-09-29T16:40:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:27 dist-test-slave-3h01 krb5kdc[7127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:27 dist-test-slave-3h01 krb5kdc[7127](info): set up 2 sockets
Sep 29 16:40:27 dist-test-slave-3h01 krb5kdc[7127](info): commencing operation
krb5kdc: starting...
W20260929 16:40:28.387511 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:28 dist-test-slave-3h01 krb5kdc[7127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:28.622026 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:28.946678 7140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:28.946990 7140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:28.947126 7140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:28.954562 7140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:28.954710 7140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:28.954792 7140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:28.954877 7140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:28.954938 7140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:28.964998 7140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:28.966334 7140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:28.967717 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:28.978895 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:28.978839 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:28.978921 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:28.979956 7140 server_base.cc:1034] running on GCE node
I20260929 16:40:28.980679 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:28.982077 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:28.983299 7140 hybrid_clock.cc:648] HybridClock initialized: now 1790700028983249 us; error 34 us; skew 500 ppm
I20260929 16:40:28.983659 7140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:28 dist-test-slave-3h01 krb5kdc[7127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:28.986697 7140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:28.988159 7140 webserver.cc:460] Webserver started at http://127.0.0.1:43523/ using document root <none> and password file <none>
I20260929 16:40:28.988937 7140 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:28.989066 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:28.989378 7140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:28.992352 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "64dc9c02850449598a5d3aba82d6b0fe"
format_stamp: "Formatted at 2026-09-29 16:40:28 on dist-test-slave-3h01"
server_key: "5c805ddd69686d2d9c467ad92b0db0cb"
server_key_iv: "b0bd628e93974ff58d0ce637592c41c5"
server_key_version: "encryptionkey@0"
I20260929 16:40:28.993114 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "64dc9c02850449598a5d3aba82d6b0fe"
format_stamp: "Formatted at 2026-09-29 16:40:28 on dist-test-slave-3h01"
server_key: "5c805ddd69686d2d9c467ad92b0db0cb"
server_key_iv: "b0bd628e93974ff58d0ce637592c41c5"
server_key_version: "encryptionkey@0"
I20260929 16:40:28.998204 7140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:40:29.001900 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:29.003337 7140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:29.003505 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "64dc9c02850449598a5d3aba82d6b0fe"
format_stamp: "Formatted at 2026-09-29 16:40:28 on dist-test-slave-3h01"
server_key: "5c805ddd69686d2d9c467ad92b0db0cb"
server_key_iv: "b0bd628e93974ff58d0ce637592c41c5"
server_key_version: "encryptionkey@0"
I20260929 16:40:29.003702 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:29.032377 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:29.036990 7140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:29.037286 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:29.055821 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33261
I20260929 16:40:29.055840 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33261 every 8 connection(s)
I20260929 16:40:29.057353 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:29.061226 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:29.063395 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7140
I20260929 16:40:29.063544 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:29.063998 18228 external_mini_cluster.cc:1428] Setting key 76aa77f743424707b66c50f301279ae1
I20260929 16:40:29.074481 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe: Bootstrap starting.
I20260929 16:40:29.078043 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe: Neither blocks nor log segments found. Creating new log.
Sep 29 16:40:29 dist-test-slave-3h01 krb5kdc[7127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:40:29.079034 7207 log.cc:824] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:29.082494 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe: No bootstrap required, opened a new log
I20260929 16:40:29.088506 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:29.065999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:49757 (local address 127.0.0.1:33261)
0929 16:40:29.066831 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:29.066857 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:29.066913 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:40:29.069201 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:29.069231 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:29.069760 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:29.070235 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:29.071609 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:29.073606 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:29.074984 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:29.075991 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:29.076197 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:29.080206 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:29.080214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:29.080229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:29.080264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:29.083981 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:29.084713 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:29.084721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:29.084730 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:29.084957 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:29.085349 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:29.085354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:29.085360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:29.086059 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:29.086364 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:29.086735 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:40:29.087082 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":338,"threads_started":1}
I20260929 16:40:29.092218 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:29.091986 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260929 16:40:29.092374 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:29.092464 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc9c02850449598a5d3aba82d6b0fe, State: Initialized, Role: FOLLOWER
I20260929 16:40:29.092998 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260929 16:40:29.093209 7207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:29.093329 7207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:29.093474 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:29.095852 7207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260929 16:40:29.096418 7207 leader_election.cc:302] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64dc9c02850449598a5d3aba82d6b0fe; no voters:
I20260929 16:40:29.097093 7207 leader_election.cc:288] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:29.097155 7217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:29.099426 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:29.099494 7217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [term 1 LEADER]: Becoming Leader. State: Replica: 64dc9c02850449598a5d3aba82d6b0fe, State: Running, Role: LEADER
I20260929 16:40:29.100015 7217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260929 16:40:29.102486 7218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64dc9c02850449598a5d3aba82d6b0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } } }
I20260929 16:40:29.102974 7218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:29.102901 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64dc9c02850449598a5d3aba82d6b0fe. Latest consensus state: current_term: 1 leader_uuid: "64dc9c02850449598a5d3aba82d6b0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64dc9c02850449598a5d3aba82d6b0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } } }
I20260929 16:40:29.103178 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:29.110006 7225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:29.115136 7225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:29.120560 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:29.096476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:41099 (local address 127.0.0.1:33261)
0929 16:40:29.096630 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:29.096636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:29.102084 (+ 5448us) server_negotiation.cc:407] Connection header received
0929 16:40:29.102371 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:29.102376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:29.102501 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:29.102584 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:29.104587 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:29.105719 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:29.107809 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:29.108552 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:29.108671 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:29.112219 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:29.112227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:29.112232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:29.112272 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:29.116259 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:29.117333 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:29.117339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:29.117343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:29.117500 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:29.118084 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:29.118090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:29.118094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:29.118552 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:29.118827 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:29.120025 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0929 16:40:29.120208 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260929 16:40:29.125262 7225 catalog_manager.cc:1347] Generated new cluster ID: 3be1a4c7d98b49df93ca0479e55c4b6b
I20260929 16:40:29.125362 7225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:29.160537 7225 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:29.162406 7225 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:29.177663 7225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64dc9c02850449598a5d3aba82d6b0fe: Generated new TSK 0
I20260929 16:40:29.178743 7225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:29.187619 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7140
I20260929 16:40:29.188767 7202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:29.282740 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7140
W20260929 16:40:29.297009 7125 connection.cc:535] client connection to 127.0.0.1:33261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33261): Connection reset by peer (error 104)
2026-09-29T16:40:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:29 dist-test-slave-3h01 krb5kdc[7239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:29 dist-test-slave-3h01 krb5kdc[7239](info): set up 2 sockets
Sep 29 16:40:29 dist-test-slave-3h01 krb5kdc[7239](info): commencing operation
krb5kdc: starting...
W20260929 16:40:30.303435 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:30 dist-test-slave-3h01 krb5kdc[7239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:30.547991 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:30.867801 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:30.868103 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:30.868244 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:30.875559 7252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:30.875686 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:30.875765 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:30.875846 7252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:30.875903 7252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:30.885856 7252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:30.887181 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:30.888533 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:30.900111 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:30.900048 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:30.900069 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:30.900781 7252 server_base.cc:1034] running on GCE node
I20260929 16:40:30.901706 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:30.903230 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:30.904475 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790700030904413 us; error 56 us; skew 500 ppm
I20260929 16:40:30.904891 7252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:30 dist-test-slave-3h01 krb5kdc[7239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:30.908041 7252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:30.909566 7252 webserver.cc:460] Webserver started at http://127.0.0.1:45293/ using document root <none> and password file <none>
I20260929 16:40:30.910279 7252 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:30.910401 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:30.910694 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:30.913818 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "73ef2d1c81f0435a822275e370c4f8cd"
format_stamp: "Formatted at 2026-09-29 16:40:30 on dist-test-slave-3h01"
server_key: "d55e967c02876b0090f65615005d685a"
server_key_iv: "42f2d3e2ffde5dba8949647d1fa37e89"
server_key_version: "encryptionkey@0"
I20260929 16:40:30.914577 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "73ef2d1c81f0435a822275e370c4f8cd"
format_stamp: "Formatted at 2026-09-29 16:40:30 on dist-test-slave-3h01"
server_key: "d55e967c02876b0090f65615005d685a"
server_key_iv: "42f2d3e2ffde5dba8949647d1fa37e89"
server_key_version: "encryptionkey@0"
I20260929 16:40:30.919845 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:40:30.923573 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:30.924983 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:30.925148 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "73ef2d1c81f0435a822275e370c4f8cd"
format_stamp: "Formatted at 2026-09-29 16:40:30 on dist-test-slave-3h01"
server_key: "d55e967c02876b0090f65615005d685a"
server_key_iv: "42f2d3e2ffde5dba8949647d1fa37e89"
server_key_version: "encryptionkey@0"
I20260929 16:40:30.925338 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:30.936496 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:30.940738 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:30.941025 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:30.958091 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40573
I20260929 16:40:30.958113 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40573 every 8 connection(s)
I20260929 16:40:30.959503 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:30.963167 7319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:30.968858 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7252
I20260929 16:40:30.969033 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:30.969539 18228 external_mini_cluster.cc:1428] Setting key ff74bc5628ad412abadc7c3f2a774270
I20260929 16:40:30.973932 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd: Bootstrap starting.
I20260929 16:40:30.977475 7319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:30.978714 7319 log.cc:824] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:30.982009 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd: No bootstrap required, opened a new log
I20260929 16:40:30.983126 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:30.971411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:57035 (local address 127.0.0.1:40573)
0929 16:40:30.972209 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:30.972221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:30.972261 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:40:30.973860 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:30.973882 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:30.974318 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:30.974649 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:30.975260 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:30.976607 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:30.978461 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:30.979274 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:30.979473 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:30.980501 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:30.980512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:30.980529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:30.981218 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:30.981312 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:30.981349 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:30.981507 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:30.981655 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:40:30.981827 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":362,"threads_started":1}
I20260929 16:40:30.987084 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:30.991954 7319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260929 16:40:30.992393 7319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:30.992491 7319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ef2d1c81f0435a822275e370c4f8cd, State: Initialized, Role: FOLLOWER
I20260929 16:40:30.993156 7319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260929 16:40:30.993392 7319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:30.993539 7319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:30.993731 7319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:30.996431 7319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260929 16:40:30.997009 7319 leader_election.cc:302] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ef2d1c81f0435a822275e370c4f8cd; no voters:
I20260929 16:40:30.997633 7319 leader_election.cc:288] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:30.997750 7329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:30.998914 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:30.992033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:44305 (local address 127.0.0.1:40573)
0929 16:40:30.992213 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:30.992218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:30.992659 (+ 441us) server_negotiation.cc:407] Connection header received
0929 16:40:30.992845 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:30.992850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:30.992929 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:30.993030 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:30.993464 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:30.994536 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:30.995871 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:30.996535 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:30.996654 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:30.997694 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:30.997700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:30.997705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:30.997732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:30.997797 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:30.997820 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:30.997938 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:30.998498 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 16:40:30.998560 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:40:30.999774 7319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:30.999811 7329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [term 1 LEADER]: Becoming Leader. State: Replica: 73ef2d1c81f0435a822275e370c4f8cd, State: Running, Role: LEADER
I20260929 16:40:31.000397 7329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260929 16:40:31.003296 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73ef2d1c81f0435a822275e370c4f8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } } }
I20260929 16:40:31.003712 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:31.003696 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73ef2d1c81f0435a822275e370c4f8cd. Latest consensus state: current_term: 1 leader_uuid: "73ef2d1c81f0435a822275e370c4f8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ef2d1c81f0435a822275e370c4f8cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } } }
I20260929 16:40:31.003966 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:31.004627 7339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:31.009498 7339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:31.017791 7339 catalog_manager.cc:1347] Generated new cluster ID: 3cf88e8e407e489b949b8c8391a034ff
I20260929 16:40:31.017915 7339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:31.042989 7339 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:31.044976 7339 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:31.057621 7339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73ef2d1c81f0435a822275e370c4f8cd: Generated new TSK 0
I20260929 16:40:31.058735 7339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:31.065696 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7252
W20260929 16:40:31.067009 7314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:57035
W20260929 16:40:31.067795 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:31.067939 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7252
W20260929 16:40:31.081633 7237 connection.cc:535] client connection to 127.0.0.1:40573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40573): Connection reset by peer (error 104)
2026-09-29T16:40:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1783 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:31 dist-test-slave-3h01 krb5kdc[7350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:31 dist-test-slave-3h01 krb5kdc[7350](info): set up 2 sockets
Sep 29 16:40:31 dist-test-slave-3h01 krb5kdc[7350](info): commencing operation
krb5kdc: starting...
W20260929 16:40:32.083468 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:32 dist-test-slave-3h01 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:40:32.317852 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:32.650607 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:32.650928 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:32.651077 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:32.658445 7363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:32.658589 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:32.658692 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:32.658787 7363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:32.658856 7363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:32.669122 7363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:32.670502 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:32.671986 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:32.683969 7368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:32.684041 7369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:32.683962 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:32.684746 7363 server_base.cc:1034] running on GCE node
I20260929 16:40:32.685546 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:32.686990 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:32.688189 7363 hybrid_clock.cc:648] HybridClock initialized: now 1790700032688132 us; error 49 us; skew 500 ppm
I20260929 16:40:32.688545 7363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:32 dist-test-slave-3h01 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:32.694676 7363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:40:32.696236 7363 webserver.cc:460] Webserver started at http://127.0.0.1:39219/ using document root <none> and password file <none>
I20260929 16:40:32.696950 7363 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:32.697070 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:32.697365 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:32.700455 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "21b810add3324aac92acc3a6aa406994"
format_stamp: "Formatted at 2026-09-29 16:40:32 on dist-test-slave-3h01"
server_key: "61ee1efe0f303541151cae67bef85f22"
server_key_iv: "06b355006d18afd3e86a1b1a4a0627ac"
server_key_version: "encryptionkey@0"
I20260929 16:40:32.701196 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "21b810add3324aac92acc3a6aa406994"
format_stamp: "Formatted at 2026-09-29 16:40:32 on dist-test-slave-3h01"
server_key: "61ee1efe0f303541151cae67bef85f22"
server_key_iv: "06b355006d18afd3e86a1b1a4a0627ac"
server_key_version: "encryptionkey@0"
I20260929 16:40:32.706374 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:40:32.709910 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:32.711280 7363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:40:32.711447 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "21b810add3324aac92acc3a6aa406994"
format_stamp: "Formatted at 2026-09-29 16:40:32 on dist-test-slave-3h01"
server_key: "61ee1efe0f303541151cae67bef85f22"
server_key_iv: "06b355006d18afd3e86a1b1a4a0627ac"
server_key_version: "encryptionkey@0"
I20260929 16:40:32.711620 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:32.717124 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:32.721133 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:32.721406 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:32.738265 7363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42051
I20260929 16:40:32.738313 7429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42051 every 8 connection(s)
I20260929 16:40:32.739655 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:32.743320 7430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:32.748898 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7363
I20260929 16:40:32.749037 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:32.749436 18228 external_mini_cluster.cc:1428] Setting key 4bc434d4251a1f6b3f36844d94d27508
I20260929 16:40:32.754837 7430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994: Bootstrap starting.
I20260929 16:40:32.758343 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:32.751292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:44411 (local address 127.0.0.1:42051)
0929 16:40:32.752131 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:32.752151 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:32.752202 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:40:32.754500 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:32.754531 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:32.754983 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:32.755461 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:32.756425 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:32.756434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:32.756457 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:32.756475 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:32.756549 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:32.756581 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:32.756811 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:32.756939 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 16:40:32.757141 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
I20260929 16:40:32.758837 7430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:32.759924 7430 log.cc:824] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:32.761968 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:32.762806 7430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994: No bootstrap required, opened a new log
I20260929 16:40:32.768353 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:32.766106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:33229 (local address 127.0.0.1:42051)
0929 16:40:32.766288 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:32.766293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:32.766604 (+ 311us) server_negotiation.cc:407] Connection header received
0929 16:40:32.766730 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:32.766736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:32.766797 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:32.766910 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:32.767750 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:32.767755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:32.767759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:32.767777 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:32.767827 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:32.767847 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:32.767970 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:32.768015 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 16:40:32.768074 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:40:32.770872 7430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260929 16:40:32.771238 7430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:32.771333 7430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b810add3324aac92acc3a6aa406994, State: Initialized, Role: FOLLOWER
I20260929 16:40:32.771883 7430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260929 16:40:32.772099 7430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:32.772193 7430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:32.772317 7430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:32.774423 7430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260929 16:40:32.774901 7430 leader_election.cc:302] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21b810add3324aac92acc3a6aa406994; no voters:
I20260929 16:40:32.775454 7430 leader_election.cc:288] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:32.775583 7440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:32.776185 7440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [term 1 LEADER]: Becoming Leader. State: Replica: 21b810add3324aac92acc3a6aa406994, State: Running, Role: LEADER
I20260929 16:40:32.776728 7440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260929 16:40:32.777407 7430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:32.779529 7442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b810add3324aac92acc3a6aa406994. Latest consensus state: current_term: 1 leader_uuid: "21b810add3324aac92acc3a6aa406994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260929 16:40:32.779975 7442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:32.780488 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21b810add3324aac92acc3a6aa406994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b810add3324aac92acc3a6aa406994" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260929 16:40:32.780746 7449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:32.780764 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:32.785665 7449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:32.798219 7449 catalog_manager.cc:1347] Generated new cluster ID: 408738f8c58049e68eabbf826a92ec86
I20260929 16:40:32.798342 7449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:32.811398 7449 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:32.811792 7449 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:32.816911 7449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21b810add3324aac92acc3a6aa406994: Generated new TSK 0
I20260929 16:40:32.817670 7449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:32.830490 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7363
W20260929 16:40:32.831727 7425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:44411
W20260929 16:40:32.832422 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:32.832547 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7363
2026-09-29T16:40:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1764 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:32 dist-test-slave-3h01 krb5kdc[7461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:32 dist-test-slave-3h01 krb5kdc[7461](info): set up 2 sockets
Sep 29 16:40:32 dist-test-slave-3h01 krb5kdc[7461](info): commencing operation
krb5kdc: starting...
W20260929 16:40:33.835419 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:33 dist-test-slave-3h01 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:34.079056 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36517
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:34.404058 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:34.404356 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:34.404487 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:34.411761 7474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:34.411877 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:34.411929 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:34.412011 7474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:34.412068 7474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:34.421983 7474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:36517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:34.423336 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:34.424650 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:34.436169 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:34.436206 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:34.436232 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:34.437388 7474 server_base.cc:1034] running on GCE node
I20260929 16:40:34.438256 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:34.439815 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:34.441022 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790700034440969 us; error 43 us; skew 500 ppm
I20260929 16:40:34.441380 7474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:34 dist-test-slave-3h01 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700034, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:34.444535 7474 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:34.446079 7474 webserver.cc:460] Webserver started at http://127.17.205.62:36069/ using document root <none> and password file <none>
I20260929 16:40:34.446775 7474 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:34.446890 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:34.447176 7474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:34.450193 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "972f49488c9048b693dc37cef89f80bc"
format_stamp: "Formatted at 2026-09-29 16:40:34 on dist-test-slave-3h01"
server_key: "cb4d8a4743ea949b7e3552c9e7813230"
server_key_iv: "632be60c8ccb6e88a8bae5b589d20ef8"
server_key_version: "encryptionkey@0"
I20260929 16:40:34.450954 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "972f49488c9048b693dc37cef89f80bc"
format_stamp: "Formatted at 2026-09-29 16:40:34 on dist-test-slave-3h01"
server_key: "cb4d8a4743ea949b7e3552c9e7813230"
server_key_iv: "632be60c8ccb6e88a8bae5b589d20ef8"
server_key_version: "encryptionkey@0"
I20260929 16:40:34.456261 7474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:40:34.459888 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:34.461411 7474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:34.461606 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "972f49488c9048b693dc37cef89f80bc"
format_stamp: "Formatted at 2026-09-29 16:40:34 on dist-test-slave-3h01"
server_key: "cb4d8a4743ea949b7e3552c9e7813230"
server_key_iv: "632be60c8ccb6e88a8bae5b589d20ef8"
server_key_version: "encryptionkey@0"
I20260929 16:40:34.461803 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:34.488407 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:34.492825 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:34.493106 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:34.510558 7474 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:36517
I20260929 16:40:34.510571 7540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:36517 every 8 connection(s)
I20260929 16:40:34.512007 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:34.512645 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7474
I20260929 16:40:34.512840 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:34.513358 18228 external_mini_cluster.cc:1428] Setting key e167a06d69c0beb1541f78e3cdab181a
I20260929 16:40:34.516808 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:40:34 dist-test-slave-3h01 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:34.529691 7541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc: Bootstrap starting.
I20260929 16:40:34.533932 7541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:34.535118 7541 log.cc:824] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:34.538509 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:34.515168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35467 (local address 127.17.205.62:36517)
0929 16:40:34.515997 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:34.516013 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:34.516070 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:40:34.518289 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:34.518318 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:34.518841 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:34.519332 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:34.521037 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:34.523102 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:34.524386 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:34.525433 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:34.525619 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:34.529748 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:34.529759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:34.529781 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:34.529831 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:34.533615 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:34.534500 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:34.534508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:34.534517 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:34.534719 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:34.535235 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:34.535241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:34.535244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:34.535963 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:34.536209 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:34.536825 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 16:40:34.537156 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":307,"threads_started":1}
I20260929 16:40:34.538640 7541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc: No bootstrap required, opened a new log
I20260929 16:40:34.542586 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:34.546778 7541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } }
I20260929 16:40:34.547169 7541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:34.547293 7541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972f49488c9048b693dc37cef89f80bc, State: Initialized, Role: FOLLOWER
I20260929 16:40:34.547925 7541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } }
I20260929 16:40:34.548143 7541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:34.548278 7541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:34.548432 7541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:34.551110 7541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } }
I20260929 16:40:34.551700 7541 leader_election.cc:302] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972f49488c9048b693dc37cef89f80bc; no voters:
I20260929 16:40:34.552385 7541 leader_election.cc:288] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:34.552500 7551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:34.554247 7541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:34.555444 7551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [term 1 LEADER]: Becoming Leader. State: Replica: 972f49488c9048b693dc37cef89f80bc, State: Running, Role: LEADER
I20260929 16:40:34.556655 7551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } }
I20260929 16:40:34.559419 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "972f49488c9048b693dc37cef89f80bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } } }
I20260929 16:40:34.559396 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 972f49488c9048b693dc37cef89f80bc. Latest consensus state: current_term: 1 leader_uuid: "972f49488c9048b693dc37cef89f80bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f49488c9048b693dc37cef89f80bc" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36517 } } }
I20260929 16:40:34.559840 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:34.559840 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:34.560729 7561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:34.563318 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:34.546712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57965 (local address 127.17.205.62:36517)
0929 16:40:34.546904 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:34.546910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:34.547176 (+ 266us) server_negotiation.cc:407] Connection header received
0929 16:40:34.547453 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:34.547458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:34.547537 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:34.547637 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:34.549132 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:34.550265 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:34.551459 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:34.552094 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:34.552227 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:34.555312 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:34.555319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:34.555325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:34.555369 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:34.559057 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:34.560082 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:34.560088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:34.560092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:34.560292 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:34.560877 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:34.560886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:34.560890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:34.561276 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:34.562540 (+ 1264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:34.562709 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:40:34.562912 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:40:34.571820 7561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:34.581948 7561 catalog_manager.cc:1347] Generated new cluster ID: 7e68d57eea6a4f96a1a0007effd97e60
I20260929 16:40:34.582059 7561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:34.608340 7561 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:34.609627 7561 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:34.621796 7561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 972f49488c9048b693dc37cef89f80bc: Generated new TSK 0
I20260929 16:40:34.622912 7561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:34.636232 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7474
I20260929 16:40:34.638221 7536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:34.740939 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7474
W20260929 16:40:34.755065 7459 connection.cc:535] client connection to 127.17.205.62:36517 recv error: Network error: failed to read from TLS socket (remote: 127.17.205.62:36517): Connection reset by peer (error 104)
2026-09-29T16:40:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:34 dist-test-slave-3h01 krb5kdc[7573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:34 dist-test-slave-3h01 krb5kdc[7573](info): set up 2 sockets
Sep 29 16:40:34 dist-test-slave-3h01 krb5kdc[7573](info): commencing operation
krb5kdc: starting...
W20260929 16:40:35.751736 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:35 dist-test-slave-3h01 krb5kdc[7573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:35.998966 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45953
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:45953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:36.321394 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:36.321672 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:36.321789 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:36.328961 7586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:36.329077 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:36.329129 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:36.329187 7586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:36.329232 7586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:36.338935 7586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:45953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:36.340216 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:36.341617 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:36.353386 7591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:36.353418 7592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:36.353333 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:36.354581 7586 server_base.cc:1034] running on GCE node
I20260929 16:40:36.355523 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:36.357125 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:36.358361 7586 hybrid_clock.cc:648] HybridClock initialized: now 1790700036358309 us; error 43 us; skew 500 ppm
I20260929 16:40:36.358723 7586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:36 dist-test-slave-3h01 krb5kdc[7573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700036, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:36.361873 7586 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:36.363477 7586 webserver.cc:460] Webserver started at http://127.17.205.62:42911/ using document root <none> and password file <none>
I20260929 16:40:36.364197 7586 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:36.364316 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:36.364604 7586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:36.367684 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "b94e035f3afb4808ad3ada978f9ce165"
format_stamp: "Formatted at 2026-09-29 16:40:36 on dist-test-slave-3h01"
server_key: "a180a60a194f03d0f39dcd6e3e1bce12"
server_key_iv: "c2a9520ce8e468fd761483d63a071909"
server_key_version: "encryptionkey@0"
I20260929 16:40:36.368431 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "b94e035f3afb4808ad3ada978f9ce165"
format_stamp: "Formatted at 2026-09-29 16:40:36 on dist-test-slave-3h01"
server_key: "a180a60a194f03d0f39dcd6e3e1bce12"
server_key_iv: "c2a9520ce8e468fd761483d63a071909"
server_key_version: "encryptionkey@0"
I20260929 16:40:36.373615 7586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 16:40:36.377136 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:36.378479 7586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:40:36.378648 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "b94e035f3afb4808ad3ada978f9ce165"
format_stamp: "Formatted at 2026-09-29 16:40:36 on dist-test-slave-3h01"
server_key: "a180a60a194f03d0f39dcd6e3e1bce12"
server_key_iv: "c2a9520ce8e468fd761483d63a071909"
server_key_version: "encryptionkey@0"
I20260929 16:40:36.378839 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:36.397606 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:36.401804 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:36.402081 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:36.419159 7586 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:45953
I20260929 16:40:36.419171 7652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:45953 every 8 connection(s)
I20260929 16:40:36.420548 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:36.422035 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7586
I20260929 16:40:36.422195 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:36.422639 18228 external_mini_cluster.cc:1428] Setting key 8baa8c20336529fad9b7e7441431e438
I20260929 16:40:36.424816 7653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:36.437325 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:36.424451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34071 (local address 127.17.205.62:45953)
0929 16:40:36.425198 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:36.425214 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:36.425274 (+ 60us) server_negotiation.cc:407] Connection header received
0929 16:40:36.427499 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:36.427529 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:36.428058 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:36.428519 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:36.429025 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:36.431037 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:36.432313 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:36.433328 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:36.433499 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:36.434555 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:36.434569 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:36.434587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:36.435330 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:36.435421 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:36.435461 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:36.435624 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:36.435842 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 16:40:36.436023 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260929 16:40:36.438553 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165: Bootstrap starting.
I20260929 16:40:36.441246 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:36.441913 7653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:36.443120 7653 log.cc:824] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:36.445879 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165: No bootstrap required, opened a new log
I20260929 16:40:36.452579 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:36.446162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43367 (local address 127.17.205.62:45953)
0929 16:40:36.446346 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:36.446352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:36.446628 (+ 276us) server_negotiation.cc:407] Connection header received
0929 16:40:36.446813 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:36.446816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:36.446871 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:36.446945 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:36.447533 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:36.448325 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:36.450135 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:36.450565 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:36.450661 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:36.451871 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:36.451876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:36.451879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:36.451895 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:36.451935 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:36.451953 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:36.452031 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:36.452210 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:40:36.452262 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:40:36.454870 7653 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } }
I20260929 16:40:36.455248 7653 raft_consensus.cc:374] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:36.455338 7653 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94e035f3afb4808ad3ada978f9ce165, State: Initialized, Role: FOLLOWER
I20260929 16:40:36.455928 7653 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } }
I20260929 16:40:36.456146 7653 raft_consensus.cc:388] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:36.456275 7653 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:36.456422 7653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:36.458520 7653 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } }
I20260929 16:40:36.459007 7653 leader_election.cc:302] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b94e035f3afb4808ad3ada978f9ce165; no voters:
I20260929 16:40:36.459591 7653 leader_election.cc:288] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:36.459700 7663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:36.460355 7663 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [term 1 LEADER]: Becoming Leader. State: Replica: b94e035f3afb4808ad3ada978f9ce165, State: Running, Role: LEADER
I20260929 16:40:36.460894 7663 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } }
I20260929 16:40:36.461580 7653 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:36.463840 7664 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b94e035f3afb4808ad3ada978f9ce165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } } }
I20260929 16:40:36.463821 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94e035f3afb4808ad3ada978f9ce165. Latest consensus state: current_term: 1 leader_uuid: "b94e035f3afb4808ad3ada978f9ce165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94e035f3afb4808ad3ada978f9ce165" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45953 } } }
I20260929 16:40:36.464231 7664 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:36.464282 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:36.465827 7670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:36.471115 7670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:36.479785 7670 catalog_manager.cc:1347] Generated new cluster ID: dea1c3500f9a4ec88b7bb6b54e118be2
I20260929 16:40:36.479890 7670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:36.493678 7670 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:36.495033 7670 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:36.506824 7670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94e035f3afb4808ad3ada978f9ce165: Generated new TSK 0
I20260929 16:40:36.507723 7670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:36.516302 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7586
W20260929 16:40:36.517803 7648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34071
W20260929 16:40:36.518546 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:36.518679 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7586
W20260929 16:40:36.532414 7571 connection.cc:535] client connection to 127.17.205.62:45953 recv error: Network error: failed to read from TLS socket (remote: 127.17.205.62:45953): Connection reset by peer (error 104)
2026-09-29T16:40:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:36 dist-test-slave-3h01 krb5kdc[7684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:36 dist-test-slave-3h01 krb5kdc[7684](info): set up 2 sockets
Sep 29 16:40:36 dist-test-slave-3h01 krb5kdc[7684](info): commencing operation
krb5kdc: starting...
W20260929 16:40:37.527489 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:37 dist-test-slave-3h01 krb5kdc[7684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:37.767954 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42011
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:42011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:38.087587 7697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:38.087885 7697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:38.088023 7697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:38.095299 7697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:38.095415 7697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:38.095489 7697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:38.095569 7697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:38.095629 7697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:38.105734 7697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:42011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:42011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:38.107034 7697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:38.108358 7697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:38.119649 7702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:38.119581 7705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:38.119616 7703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:38.120282 7697 server_base.cc:1034] running on GCE node
I20260929 16:40:38.121061 7697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:38.122478 7697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:38.123665 7697 hybrid_clock.cc:648] HybridClock initialized: now 1790700038123607 us; error 56 us; skew 500 ppm
I20260929 16:40:38.123998 7697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:38 dist-test-slave-3h01 krb5kdc[7684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700038, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:38.129874 7697 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:38.131485 7697 webserver.cc:460] Webserver started at http://127.17.205.62:46325/ using document root <none> and password file <none>
I20260929 16:40:38.132181 7697 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:38.132299 7697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:38.132591 7697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:38.135720 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "c462bcb0e2834e94894e1b13f9f9b340"
format_stamp: "Formatted at 2026-09-29 16:40:38 on dist-test-slave-3h01"
server_key: "3891d990b24aea21779e047d115d9660"
server_key_iv: "31c09edcf1fc104b84ac9731984d5bbb"
server_key_version: "encryptionkey@0"
I20260929 16:40:38.136488 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "c462bcb0e2834e94894e1b13f9f9b340"
format_stamp: "Formatted at 2026-09-29 16:40:38 on dist-test-slave-3h01"
server_key: "3891d990b24aea21779e047d115d9660"
server_key_iv: "31c09edcf1fc104b84ac9731984d5bbb"
server_key_version: "encryptionkey@0"
I20260929 16:40:38.141857 7697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:40:38.145326 7712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:38.146698 7697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:40:38.146876 7697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "c462bcb0e2834e94894e1b13f9f9b340"
format_stamp: "Formatted at 2026-09-29 16:40:38 on dist-test-slave-3h01"
server_key: "3891d990b24aea21779e047d115d9660"
server_key_iv: "31c09edcf1fc104b84ac9731984d5bbb"
server_key_version: "encryptionkey@0"
I20260929 16:40:38.147066 7697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:38.152697 7697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:38.156549 7697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:38.156813 7697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:38.173769 7697 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:42011
I20260929 16:40:38.173791 7763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:42011 every 8 connection(s)
I20260929 16:40:38.175154 7697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:38.178964 7764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:38.181201 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7697
I20260929 16:40:38.181365 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:38.181770 18228 external_mini_cluster.cc:1428] Setting key 12bbf3ba9860c00b5db42e573b77bc4a
I20260929 16:40:38.190675 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:38.183613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40109 (local address 127.17.205.62:42011)
0929 16:40:38.184409 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:38.184425 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:38.184483 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:40:38.186738 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:38.186770 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:38.187254 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:38.187707 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:38.188687 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:38.188695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:38.188713 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:38.188736 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:38.188803 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:38.188829 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:38.189051 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:38.189198 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:40:38.189398 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":323,"threads_started":1}
I20260929 16:40:38.192157 7764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340: Bootstrap starting.
I20260929 16:40:38.194216 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:38.195327 7764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:38.196286 7764 log.cc:824] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:38.198866 7764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340: No bootstrap required, opened a new log
I20260929 16:40:38.203084 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:38.200701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49377 (local address 127.17.205.62:42011)
0929 16:40:38.200904 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:38.200912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:38.201276 (+ 364us) server_negotiation.cc:407] Connection header received
0929 16:40:38.201460 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:38.201464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:38.201522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:38.201601 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:38.202496 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:38.202500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:38.202503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:38.202517 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:38.202555 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:38.202570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:38.202665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:38.202770 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:40:38.202818 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:40:38.207378 7764 raft_consensus.cc:348] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } }
I20260929 16:40:38.207774 7764 raft_consensus.cc:374] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:38.207849 7764 raft_consensus.cc:729] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c462bcb0e2834e94894e1b13f9f9b340, State: Initialized, Role: FOLLOWER
I20260929 16:40:38.208397 7764 consensus_queue.cc:260] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } }
I20260929 16:40:38.208602 7764 raft_consensus.cc:388] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:38.208710 7764 raft_consensus.cc:482] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:38.208835 7764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:38.210923 7764 raft_consensus.cc:504] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } }
I20260929 16:40:38.211421 7764 leader_election.cc:302] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c462bcb0e2834e94894e1b13f9f9b340; no voters:
I20260929 16:40:38.211977 7764 leader_election.cc:288] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:38.212129 7774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:38.212875 7774 raft_consensus.cc:686] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [term 1 LEADER]: Becoming Leader. State: Replica: c462bcb0e2834e94894e1b13f9f9b340, State: Running, Role: LEADER
I20260929 16:40:38.213467 7774 consensus_queue.cc:237] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } }
I20260929 16:40:38.214054 7764 sys_catalog.cc:564] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:38.216187 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c462bcb0e2834e94894e1b13f9f9b340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } } }
I20260929 16:40:38.216256 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c462bcb0e2834e94894e1b13f9f9b340. Latest consensus state: current_term: 1 leader_uuid: "c462bcb0e2834e94894e1b13f9f9b340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c462bcb0e2834e94894e1b13f9f9b340" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 42011 } } }
I20260929 16:40:38.216631 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:38.218057 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:38.218317 7781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:38.224331 7781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:38.233875 7781 catalog_manager.cc:1347] Generated new cluster ID: 69ce34762ed54bbea81d5e8e677073da
I20260929 16:40:38.233989 7781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:38.249823 7781 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:38.250303 7781 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:38.258595 7781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c462bcb0e2834e94894e1b13f9f9b340: Generated new TSK 0
I20260929 16:40:38.259514 7781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:38.266825 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7697
W20260929 16:40:38.268079 7759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40109
W20260929 16:40:38.268808 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:38.268949 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7697
2026-09-29T16:40:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1750 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:38 dist-test-slave-3h01 krb5kdc[7795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:38 dist-test-slave-3h01 krb5kdc[7795](info): set up 2 sockets
Sep 29 16:40:38 dist-test-slave-3h01 krb5kdc[7795](info): commencing operation
krb5kdc: starting...
W20260929 16:40:39.275494 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:39 dist-test-slave-3h01 krb5kdc[7795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:39.522380 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41191
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:33255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:39.850567 7808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:39.850895 7808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:39.851038 7808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:39.858510 7808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:39.858651 7808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:39.858742 7808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:39.858826 7808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:39.858887 7808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:39.869378 7808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:33255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:41191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:39.870733 7808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:39.872114 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:39.883790 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:39.883790 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:39.883833 7813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:39.884701 7808 server_base.cc:1034] running on GCE node
I20260929 16:40:39.885514 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:39.886978 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:39.888178 7808 hybrid_clock.cc:648] HybridClock initialized: now 1790700039888110 us; error 70 us; skew 500 ppm
I20260929 16:40:39.888540 7808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:39 dist-test-slave-3h01 krb5kdc[7795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700039, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:39.891585 7808 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:39.893038 7808 webserver.cc:460] Webserver started at http://127.17.205.62:41957/ using document root <none> and password file <none>
I20260929 16:40:39.893730 7808 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:39.893850 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:39.894135 7808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:39.897161 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "47aa740a8c3c49a59134d91f2b4355f9"
format_stamp: "Formatted at 2026-09-29 16:40:39 on dist-test-slave-3h01"
server_key: "01eb0de7f0609871f8cf607edbda53ee"
server_key_iv: "92da67cf08a88485f1d21ff7d86438ea"
server_key_version: "encryptionkey@0"
I20260929 16:40:39.897964 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "47aa740a8c3c49a59134d91f2b4355f9"
format_stamp: "Formatted at 2026-09-29 16:40:39 on dist-test-slave-3h01"
server_key: "01eb0de7f0609871f8cf607edbda53ee"
server_key_iv: "92da67cf08a88485f1d21ff7d86438ea"
server_key_version: "encryptionkey@0"
I20260929 16:40:39.903237 7808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:40:39.906749 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:39.908136 7808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:39.908313 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "47aa740a8c3c49a59134d91f2b4355f9"
format_stamp: "Formatted at 2026-09-29 16:40:39 on dist-test-slave-3h01"
server_key: "01eb0de7f0609871f8cf607edbda53ee"
server_key_iv: "92da67cf08a88485f1d21ff7d86438ea"
server_key_version: "encryptionkey@0"
I20260929 16:40:39.908533 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:39.928540 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:39.933063 7808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:39.933347 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:39.950982 7808 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:41191
I20260929 16:40:39.950990 7874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:41191 every 8 connection(s)
I20260929 16:40:39.952378 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:39.955497 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7808
I20260929 16:40:39.955667 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:39.956207 18228 external_mini_cluster.cc:1428] Setting key 2bc127cdda4ab25bd2e54a54f1f079c4
I20260929 16:40:39.956635 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:40:39 dist-test-slave-3h01 krb5kdc[7795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:39.971035 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9: Bootstrap starting.
I20260929 16:40:39.975054 7875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:39.976455 7875 log.cc:824] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:39.980041 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9: No bootstrap required, opened a new log
I20260929 16:40:39.980675 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:39.958192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39209 (local address 127.17.205.62:41191)
0929 16:40:39.959094 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:39.959110 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:39.959480 (+ 370us) server_negotiation.cc:407] Connection header received
0929 16:40:39.961323 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:39.961355 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:39.961930 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:39.962409 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:39.963722 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:39.965666 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:39.966862 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:39.967975 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:39.968176 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:39.972148 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:39.972160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:39.972179 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:39.972233 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:39.976027 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:39.976758 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:39.976767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:39.976777 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:39.976993 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:39.977460 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:39.977464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:39.977466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:39.978120 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:39.978427 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:39.978782 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:40:39.979276 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":373,"threads_started":1}
I20260929 16:40:39.984391 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:39.989545 7875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } }
I20260929 16:40:39.989940 7875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:39.990043 7875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47aa740a8c3c49a59134d91f2b4355f9, State: Initialized, Role: FOLLOWER
I20260929 16:40:39.990630 7875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } }
I20260929 16:40:39.990863 7875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:39.990984 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:39.991132 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:39.993361 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } }
I20260929 16:40:39.993855 7875 leader_election.cc:302] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47aa740a8c3c49a59134d91f2b4355f9; no voters:
I20260929 16:40:39.994462 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:39.994550 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:39.996835 7875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:39.997009 7885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [term 1 LEADER]: Becoming Leader. State: Replica: 47aa740a8c3c49a59134d91f2b4355f9, State: Running, Role: LEADER
I20260929 16:40:39.997656 7885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } }
I20260929 16:40:40.002781 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:39.989388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36387 (local address 127.17.205.62:41191)
0929 16:40:39.989570 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:39.989576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:39.989999 (+ 423us) server_negotiation.cc:407] Connection header received
0929 16:40:39.990174 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:39.990179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:39.990245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:39.990326 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:39.991829 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:39.992581 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:39.993791 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:39.994289 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:39.994406 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:39.996867 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:39.996874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:39.996878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:39.996922 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:40.000154 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:40.000788 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:40.000794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:40.000798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:40.000925 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:40.001316 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:40.001322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:40.001325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:40.001679 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:40.001950 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:40.002232 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:40:40.002454 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:40:40.008764 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aa740a8c3c49a59134d91f2b4355f9. Latest consensus state: current_term: 1 leader_uuid: "47aa740a8c3c49a59134d91f2b4355f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } } }
I20260929 16:40:40.009164 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:40.016244 7899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:40.016108 7887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47aa740a8c3c49a59134d91f2b4355f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aa740a8c3c49a59134d91f2b4355f9" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 41191 } } }
I20260929 16:40:40.016530 7887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:40.020591 7899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:40.029543 7899 catalog_manager.cc:1347] Generated new cluster ID: 794cc14966554f5284dfdd035f72b83a
I20260929 16:40:40.029678 7899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:40.049948 7899 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:40.052014 7899 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:40.065477 7899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47aa740a8c3c49a59134d91f2b4355f9: Generated new TSK 0
I20260929 16:40:40.066645 7899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:40.081718 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7808
I20260929 16:40:40.082940 7870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:40.175989 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7808
2026-09-29T16:40:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:40 dist-test-slave-3h01 krb5kdc[7908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:40 dist-test-slave-3h01 krb5kdc[7908](info): set up 2 sockets
Sep 29 16:40:40 dist-test-slave-3h01 krb5kdc[7908](info): commencing operation
krb5kdc: starting...
W20260929 16:40:41.179503 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:41 dist-test-slave-3h01 krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:41.422015 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36695
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:41.781424 7921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:41.781767 7921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:41.781916 7921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:41.789320 7921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:41.789472 7921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:41.789556 7921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:41.789635 7921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:41.789702 7921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:41.800398 7921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:36695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:41.801810 7921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:41.803282 7921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:41.815227 7926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:41.815303 7929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:41.815255 7927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:41.815893 7921 server_base.cc:1034] running on GCE node
I20260929 16:40:41.816799 7921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:41.818262 7921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:41.819466 7921 hybrid_clock.cc:648] HybridClock initialized: now 1790700041819410 us; error 50 us; skew 500 ppm
I20260929 16:40:41.819813 7921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:41 dist-test-slave-3h01 krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700041, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:41.822901 7921 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:41.824419 7921 webserver.cc:460] Webserver started at http://127.17.205.62:46171/ using document root <none> and password file <none>
I20260929 16:40:41.825119 7921 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:41.825234 7921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:41.825515 7921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:41.828580 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "407dfbc222c645699cbfa82b6d25f34b"
format_stamp: "Formatted at 2026-09-29 16:40:41 on dist-test-slave-3h01"
server_key: "2d0e7433a900e05030841f6f7527cf67"
server_key_iv: "ffe9c47dfbaa845aa0943a1486f34094"
server_key_version: "encryptionkey@0"
I20260929 16:40:41.829342 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "407dfbc222c645699cbfa82b6d25f34b"
format_stamp: "Formatted at 2026-09-29 16:40:41 on dist-test-slave-3h01"
server_key: "2d0e7433a900e05030841f6f7527cf67"
server_key_iv: "ffe9c47dfbaa845aa0943a1486f34094"
server_key_version: "encryptionkey@0"
I20260929 16:40:41.834601 7921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:40:41.838135 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:41.839600 7921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:41.839766 7921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "407dfbc222c645699cbfa82b6d25f34b"
format_stamp: "Formatted at 2026-09-29 16:40:41 on dist-test-slave-3h01"
server_key: "2d0e7433a900e05030841f6f7527cf67"
server_key_iv: "ffe9c47dfbaa845aa0943a1486f34094"
server_key_version: "encryptionkey@0"
I20260929 16:40:41.839955 7921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:41.863745 7921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:41.868336 7921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:41.868634 7921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:41.885999 7921 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:36695
I20260929 16:40:41.886018 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:36695 every 8 connection(s)
I20260929 16:40:41.887445 7921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:41.891140 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:41.891856 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 7921
I20260929 16:40:41.892010 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:41.892410 18228 external_mini_cluster.cc:1428] Setting key 07245e19832aca7a1aae35455f0de54d
I20260929 16:40:41.904423 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b: Bootstrap starting.
I20260929 16:40:41.906025 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:41.894127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36973 (local address 127.17.205.62:36695)
0929 16:40:41.894876 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:41.894888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:41.894930 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:40:41.896956 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:41.896980 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:41.897420 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:41.897835 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:41.898408 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:41.899745 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:41.901572 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:41.902278 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:41.902418 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:41.903709 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:41.903718 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:41.903733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:41.904293 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:41.904370 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:41.904395 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:41.904483 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:41.904731 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:40:41.905024 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":334,"threads_started":1}
I20260929 16:40:41.908016 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:41.909286 7988 log.cc:824] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:41.909741 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:41.912264 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b: No bootstrap required, opened a new log
I20260929 16:40:41.920580 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:41.914270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55823 (local address 127.17.205.62:36695)
0929 16:40:41.914458 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:41.914464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:41.914827 (+ 363us) server_negotiation.cc:407] Connection header received
0929 16:40:41.914986 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:41.914992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:41.915080 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:41.915196 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:41.915743 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:41.916812 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:41.918070 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:41.918758 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:41.918889 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:41.919865 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:41.919875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:41.919880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:41.919903 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:41.919955 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:41.919975 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:41.920065 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:41.920161 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:40:41.920269 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:40:41.921346 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } }
I20260929 16:40:41.921860 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:41.921983 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407dfbc222c645699cbfa82b6d25f34b, State: Initialized, Role: FOLLOWER
I20260929 16:40:41.922649 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } }
I20260929 16:40:41.922881 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:41.922981 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:41.923107 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:41.925674 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } }
I20260929 16:40:41.926185 7988 leader_election.cc:302] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 407dfbc222c645699cbfa82b6d25f34b; no voters:
I20260929 16:40:41.926708 7988 leader_election.cc:288] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:41.926831 7998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:41.927508 7998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [term 1 LEADER]: Becoming Leader. State: Replica: 407dfbc222c645699cbfa82b6d25f34b, State: Running, Role: LEADER
I20260929 16:40:41.928059 7998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } }
I20260929 16:40:41.929008 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:41.931134 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407dfbc222c645699cbfa82b6d25f34b. Latest consensus state: current_term: 1 leader_uuid: "407dfbc222c645699cbfa82b6d25f34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } } }
I20260929 16:40:41.931133 7999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "407dfbc222c645699cbfa82b6d25f34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407dfbc222c645699cbfa82b6d25f34b" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 36695 } } }
I20260929 16:40:41.931649 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:41.931653 7999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:41.936669 8006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:41.941102 8006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:41.949537 8006 catalog_manager.cc:1347] Generated new cluster ID: c77e3b6236dd4701aadc03052c6565fd
I20260929 16:40:41.949664 8006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:41.958231 8006 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:41.960171 8006 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:41.969163 8006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 407dfbc222c645699cbfa82b6d25f34b: Generated new TSK 0
I20260929 16:40:41.970258 8006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:41.975687 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7921
W20260929 16:40:41.976750 7983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36973
W20260929 16:40:41.977438 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 7921: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:41.977598 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 7921
2026-09-29T16:40:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1801 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:42 dist-test-slave-3h01 krb5kdc[8019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:42 dist-test-slave-3h01 krb5kdc[8019](info): set up 2 sockets
Sep 29 16:40:42 dist-test-slave-3h01 krb5kdc[8019](info): commencing operation
krb5kdc: starting...
W20260929 16:40:42.987577 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:43 dist-test-slave-3h01 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:43.246518 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43559
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:43.586690 8032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:43.587028 8032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:43.587184 8032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:43.594792 8032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:43.594944 8032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:43.595014 8032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:43.595115 8032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:43.595187 8032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:43.605777 8032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:43559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:43.607191 8032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:43.608657 8032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:43.620889 8038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:43.620944 8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:43.620963 8040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:43.621556 8032 server_base.cc:1034] running on GCE node
I20260929 16:40:43.622402 8032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:43.623993 8032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:43.625211 8032 hybrid_clock.cc:648] HybridClock initialized: now 1790700043625161 us; error 44 us; skew 500 ppm
I20260929 16:40:43.625574 8032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:43 dist-test-slave-3h01 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:43.628787 8032 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:43.630375 8032 webserver.cc:460] Webserver started at http://127.17.205.62:38467/ using document root <none> and password file <none>
I20260929 16:40:43.631103 8032 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:43.631256 8032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:43.631565 8032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:43.634639 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "1a9ad09f00e245c89dafa1d294a562a6"
format_stamp: "Formatted at 2026-09-29 16:40:43 on dist-test-slave-3h01"
server_key: "1f8e98628dab564f215ae745a0719d5e"
server_key_iv: "b300bba07e4f2c551359c687938c3d3c"
server_key_version: "encryptionkey@0"
I20260929 16:40:43.635483 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "1a9ad09f00e245c89dafa1d294a562a6"
format_stamp: "Formatted at 2026-09-29 16:40:43 on dist-test-slave-3h01"
server_key: "1f8e98628dab564f215ae745a0719d5e"
server_key_iv: "b300bba07e4f2c551359c687938c3d3c"
server_key_version: "encryptionkey@0"
I20260929 16:40:43.640774 8032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:40:43.644263 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:43.645650 8032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:40:43.645820 8032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "1a9ad09f00e245c89dafa1d294a562a6"
format_stamp: "Formatted at 2026-09-29 16:40:43 on dist-test-slave-3h01"
server_key: "1f8e98628dab564f215ae745a0719d5e"
server_key_iv: "b300bba07e4f2c551359c687938c3d3c"
server_key_version: "encryptionkey@0"
I20260929 16:40:43.646011 8032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:43.651901 8032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:43.655895 8032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:43.656190 8032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:43.674965 8032 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:43559
I20260929 16:40:43.675006 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:43559 every 8 connection(s)
I20260929 16:40:43.676625 8032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:43.680641 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:43.680686 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8032
I20260929 16:40:43.680837 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:43.681386 18228 external_mini_cluster.cc:1428] Setting key 35a4b248a7817c650b70cd6f8a5bb774
I20260929 16:40:43.688850 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:43.683129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49577 (local address 127.17.205.62:43559)
0929 16:40:43.683983 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:43.683996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:43.684041 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:40:43.685526 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:43.685549 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:43.685908 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:43.686258 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:43.687290 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:43.687297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:43.687312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:43.687327 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:43.687397 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:43.687420 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:43.687591 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:43.687693 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 16:40:43.687898 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":344,"threads_started":1}
I20260929 16:40:43.692595 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:43.694223 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: Bootstrap starting.
I20260929 16:40:43.697674 8099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:43.698769 8099 log.cc:824] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:43.699234 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:43.696890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52005 (local address 127.17.205.62:43559)
0929 16:40:43.697076 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:43.697082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:43.697369 (+ 287us) server_negotiation.cc:407] Connection header received
0929 16:40:43.697535 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:43.697540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:43.697597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:43.697683 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:43.698590 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:43.698596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:43.698600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:40:43.698621 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:43.698661 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:40:43.698677 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:43.698781 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:43.698888 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 16:40:43.698950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:40:43.701890 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: No bootstrap required, opened a new log
I20260929 16:40:43.709792 8099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } }
I20260929 16:40:43.710311 8099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:40:43.710448 8099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9ad09f00e245c89dafa1d294a562a6, State: Initialized, Role: FOLLOWER
I20260929 16:40:43.711042 8099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } }
I20260929 16:40:43.711293 8099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:40:43.711411 8099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:40:43.711537 8099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:43.713708 8099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } }
I20260929 16:40:43.714268 8099 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9ad09f00e245c89dafa1d294a562a6; no voters:
I20260929 16:40:43.714876 8099 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:40:43.715003 8109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:43.715672 8109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9ad09f00e245c89dafa1d294a562a6, State: Running, Role: LEADER
I20260929 16:40:43.716249 8109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } }
I20260929 16:40:43.716987 8099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:43.718741 8110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9ad09f00e245c89dafa1d294a562a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } } }
I20260929 16:40:43.719111 8110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:43.719480 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9ad09f00e245c89dafa1d294a562a6. Latest consensus state: current_term: 1 leader_uuid: "1a9ad09f00e245c89dafa1d294a562a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9ad09f00e245c89dafa1d294a562a6" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 43559 } } }
I20260929 16:40:43.719838 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6 [sys.catalog]: This master's current role is: LEADER
W20260929 16:40:43.727488 8125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:43.727614 8125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:43.727761 8114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:43.729946 8114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:43.739467 8114 catalog_manager.cc:1347] Generated new cluster ID: 5848c6f3a32e479293d3acafe977510f
I20260929 16:40:43.739609 8114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:43.755347 8114 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:43.755779 8114 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:43.775550 8114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9ad09f00e245c89dafa1d294a562a6: Generated new TSK 0
I20260929 16:40:43.776466 8114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:40:43.782199 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8032
W20260929 16:40:43.783387 8094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49577
W20260929 16:40:43.784103 18228 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8032: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:40:43.784245 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8032
2026-09-29T16:40:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1806 ms)
[----------] 18 tests from AuthTokenIssuingTest (32925 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:43 dist-test-slave-3h01 krb5kdc[8130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:43 dist-test-slave-3h01 krb5kdc[8130](info): set up 2 sockets
Sep 29 16:40:43 dist-test-slave-3h01 krb5kdc[8130](info): commencing operation
krb5kdc: starting...
W20260929 16:40:44.795614 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:44 dist-test-slave-3h01 krb5kdc[8130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:45.046757 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:38185
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:45.371450 8143 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:45.371783 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:45.371857 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:45.371937 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:45.379146 8143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:45.379268 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:45.379334 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:45.379385 8143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:45.379447 8143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:45.389411 8143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:45.390718 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:45.392072 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:45.403144 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:45.403203 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:45.403144 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:45.408550 8143 server_base.cc:1034] running on GCE node
I20260929 16:40:45.409312 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:45.410708 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:45.411937 8143 hybrid_clock.cc:648] HybridClock initialized: now 1790700045411878 us; error 34 us; skew 500 ppm
I20260929 16:40:45.412245 8143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:45 dist-test-slave-3h01 krb5kdc[8130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:45.415246 8143 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:45.416795 8143 webserver.cc:460] Webserver started at http://127.17.205.62:35367/ using document root <none> and password file <none>
I20260929 16:40:45.417531 8143 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:45.417634 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:45.417956 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:45.421075 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "7055ecb98f5a479c8b79bb693790173a"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "7dcb54a1c133057e32fdfaa9af65ee49"
server_key_iv: "34b726e47519c6021d2f6c8f9de12087"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.421887 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "7055ecb98f5a479c8b79bb693790173a"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "7dcb54a1c133057e32fdfaa9af65ee49"
server_key_iv: "34b726e47519c6021d2f6c8f9de12087"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.427011 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:40:45.430614 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:45.432150 8143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:45.432332 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "7055ecb98f5a479c8b79bb693790173a"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "7dcb54a1c133057e32fdfaa9af65ee49"
server_key_iv: "34b726e47519c6021d2f6c8f9de12087"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.432549 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:45.454979 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:45.459167 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:45.459458 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:45.477201 8143 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:38185
I20260929 16:40:45.477233 8209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:38185 every 8 connection(s)
I20260929 16:40:45.478557 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:45.479609 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8143
I20260929 16:40:45.479877 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:45.480767 18228 external_mini_cluster.cc:1428] Setting key 57e17e8beb192f5418d7d083854fc463
I20260929 16:40:45.483494 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:45.492583 8210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.205.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.61@KRBTEST.COM" created.
Sep 29 16:40:45 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:45.518602 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.496019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.17.205.62:38185)
0929 16:40:45.497924 (+ 1905us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:45.497931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:45.497975 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:40:45.499771 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:45.499778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:45.500028 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:45.500213 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:45.501684 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.502915 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.504924 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.505465 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.505604 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:45.511566 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:45.511575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:45.511588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:45.511622 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:45.514276 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:45.514963 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:45.514972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:45.514976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:45.515131 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:45.515547 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:45.515552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:45.515555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:45.516159 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:45.516349 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:45.517014 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 16:40:45.517402 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":1693,"spinlock_wait_cycles":15744,"thread_start_us":1436,"threads_started":1}
I20260929 16:40:45.518639 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.496019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38185 (local address 127.0.0.1:44276)
0929 16:40:45.496925 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.496977 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:45.499373 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:45.500255 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:45.500262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:45.501025 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:45.501453 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:45.501475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.503064 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.503069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:45.504789 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:45.504805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.505704 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.505712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:45.505898 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:45.506539 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:45.506575 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:45.511338 (+ 4763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:45.514411 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:45.514423 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:45.514436 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:45.514824 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:45.515273 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:45.515278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:45.515282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:45.515439 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:45.516314 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:45.516322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:45.516588 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:40:45.516870 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 16:40:45.517573 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":164,"thread_start_us":386,"threads_started":1}
I20260929 16:40:45.521679 8210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:45.522804 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.522392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49646)
0929 16:40:45.522575 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.522687 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:40:45.523106 8161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
I20260929 16:40:45.524636 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.524300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49660)
0929 16:40:45.524435 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.524503 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:40:45.524833 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:35311: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
I20260929 16:40:45.539762 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:35311
--webserver_interface=127.17.205.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:40:45.574154 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } attempt: 1
I20260929 16:40:45.575354 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.574821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49662)
0929 16:40:45.575047 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.575173 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:40:45.576427 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.576051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49668)
0929 16:40:45.576219 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.576287 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:40:45.576628 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:35311: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
I20260929 16:40:45.644872 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } attempt: 2
I20260929 16:40:45.646034 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.645569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49674)
0929 16:40:45.645794 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.645890 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:40:45.647010 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.646679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49690)
0929 16:40:45.646809 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.646880 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:40:45.647243 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:35311: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
I20260929 16:40:45.738467 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } attempt: 3
I20260929 16:40:45.739663 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.739174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49700)
0929 16:40:45.739436 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.739527 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:40:45.740644 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.740283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49708)
0929 16:40:45.740433 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.740507 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:45.740815 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:35311: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
W20260929 16:40:45.871171 8217 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:45.871585 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:45.871687 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:45.871814 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:45.879077 8217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:45.879192 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:45.879302 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:45.879380 8217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:45.879464 8217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:45.884061 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } attempt: 4
I20260929 16:40:45.885257 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.884675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49710)
0929 16:40:45.885018 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.885107 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260929 16:40:45.886219 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.885862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49716)
0929 16:40:45.886003 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.886087 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:40:45.886389 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:35311: Network error: Client connection negotiation failed: client connection to 127.17.205.61:35311: connect: Connection refused (error 111)
I20260929 16:40:45.890235 8217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.205.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:45.891634 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:45.892927 8217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:45.904418 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:45.904397 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:45.904830 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:45.905117 8217 server_base.cc:1034] running on GCE node
I20260929 16:40:45.905893 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:45.907385 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:45.908596 8217 hybrid_clock.cc:648] HybridClock initialized: now 1790700045908522 us; error 56 us; skew 500 ppm
I20260929 16:40:45.908910 8217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:45 dist-test-slave-3h01 krb5kdc[8130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:45.911998 8217 init.cc:377] Logged in from keytab as kudu/127.17.205.61@KRBTEST.COM (short username kudu)
I20260929 16:40:45.913512 8217 webserver.cc:460] Webserver started at http://127.17.205.61:41739/ using document root <none> and password file <none>
I20260929 16:40:45.914173 8217 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:45.914256 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:45.914527 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:45.917589 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data/instance:
uuid: "8a9e7ba61771426cbceed38babf9157e"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "3fc963f468e799ad8ae9823f3a182146"
server_key_iv: "44d6823cede8eee7cb2e503233083674"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.918308 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal/instance:
uuid: "8a9e7ba61771426cbceed38babf9157e"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "3fc963f468e799ad8ae9823f3a182146"
server_key_iv: "44d6823cede8eee7cb2e503233083674"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.923478 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 16:40:45.926795 8232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:45.928115 8217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:40:45.928253 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "8a9e7ba61771426cbceed38babf9157e"
format_stamp: "Formatted at 2026-09-29 16:40:45 on dist-test-slave-3h01"
server_key: "3fc963f468e799ad8ae9823f3a182146"
server_key_iv: "44d6823cede8eee7cb2e503233083674"
server_key_version: "encryptionkey@0"
I20260929 16:40:45.928443 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:45.946800 8217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:45.950623 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:45.950842 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:45.968176 8217 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.61:35311
I20260929 16:40:45.968205 8283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.61:35311 every 8 connection(s)
I20260929 16:40:45.969532 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
I20260929 16:40:45.973165 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:45.973680 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8217
I20260929 16:40:45.973868 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-1/wal/instance
I20260929 16:40:45.974390 18228 external_mini_cluster.cc:1428] Setting key 15e349de42cdb387a0c3a81510320b6c
I20260929 16:40:45.981909 8284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:45 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.60@KRBTEST.COM" created.
I20260929 16:40:46.002358 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.985207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.17.205.62:38185)
0929 16:40:45.985438 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:45.985445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:45.986533 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 16:40:45.988479 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:45.988484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:45.988565 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:45.988653 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:45.990150 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.990823 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.992355 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.992797 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.992911 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:45.996635 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:45.996640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:45.996644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:45.996679 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:45.999504 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.000211 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.000217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.000221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.000352 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.000683 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.000686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.000688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.001014 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.001145 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.001807 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.002031 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:40:46.003312 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:45.985576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38185 (local address 127.0.0.1:44292)
0929 16:40:45.986356 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:45.986405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:45.988208 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:45.988904 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:45.988913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:45.989415 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:45.989971 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:45.989989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.990971 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.990975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:45.992215 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:45.992227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:45.993020 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:45.993027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:45.993249 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:45.993884 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:45.993907 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:45.996385 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:45.999624 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:45.999632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:45.999646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.000082 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.000437 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.000440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.000443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.000605 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.001110 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.001115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.001435 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.001698 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.002146 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260929 16:40:46.005154 8284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:46.019627 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.005943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49726)
0929 16:40:46.006100 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.006120 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.006239 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.007478 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.007482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.007860 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.008015 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.008023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.009748 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.009752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.010593 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.010603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.011440 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.011446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.011623 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.012145 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.012165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.014352 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.017297 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.017303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.017307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.017643 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.017980 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.017985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.017989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.018117 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.018698 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.018702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.018830 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.018951 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.019243 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:40:46.019716 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.006225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.17.205.61:35311)
0929 16:40:46.006950 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.006957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.006988 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:40:46.007039 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.007048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.007316 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.007453 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.008187 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.009599 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.010755 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.011314 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.011420 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.014541 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.014551 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.014565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.014597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.017183 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.017756 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.017762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.017764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.017877 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.018208 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.018213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.018216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.018571 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.018752 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.019082 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.019348 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":537,"thread_start_us":362,"threads_started":1}
I20260929 16:40:46.022234 8284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:46.023386 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.022919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37620)
0929 16:40:46.023075 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.023184 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:40:46.023651 8235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.025108 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.024812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37628)
0929 16:40:46.024940 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.025006 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:40:46.025319 8284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
I20260929 16:40:46.033548 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:41641
--webserver_interface=127.17.205.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:40:46.074630 8284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 1
I20260929 16:40:46.075776 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.075354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37634)
0929 16:40:46.075535 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.075634 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:40:46.076721 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.076396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37642)
0929 16:40:46.076544 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.076617 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:46.076923 8284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.145118 8284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 2
I20260929 16:40:46.146286 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.145855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37654)
0929 16:40:46.146039 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.146131 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:40:46.147245 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.146909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37658)
0929 16:40:46.147037 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.147104 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:40:46.147432 8284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.185672 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } attempt: 5
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:46.198554 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.186577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.17.205.61:35311)
0929 16:40:46.186774 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.186780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.186802 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:40:46.187013 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.187017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.187084 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.187170 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.187992 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.188673 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.189791 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.190201 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.190318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.193538 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.193543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.193547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.193579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.196065 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.196885 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.196888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.196890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.197007 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.197370 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.197373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.197375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.197640 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.197746 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.198167 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.198335 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:40:46.198698 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.186454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49734)
0929 16:40:46.186635 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.186656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.186867 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.187250 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.187255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.187662 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.187837 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.187846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.188804 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.188810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.189644 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.189656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.190344 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.190350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.190551 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.191130 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.191148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.193338 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.196220 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.196226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.196230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.196773 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.197105 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.197110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.197114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.197283 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.197741 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.197746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.197936 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.198076 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.198360 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:40:46.199898 8210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:46.200938 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.200506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37672)
0929 16:40:46.200675 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.200776 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:40:46.201894 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.201482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37682)
0929 16:40:46.201621 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.201723 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 16:40:46.202121 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.238613 8284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 3
I20260929 16:40:46.239779 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.239312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37684)
0929 16:40:46.239523 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.239625 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:40:46.240772 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.240375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37692)
0929 16:40:46.240585 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.240660 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260929 16:40:46.240962 8284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.253314 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 1
I20260929 16:40:46.254441 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.253916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37694)
0929 16:40:46.254133 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.254244 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:40:46.255438 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.255058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37702)
0929 16:40:46.255193 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.255305 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 16:40:46.255662 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.323864 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 2
I20260929 16:40:46.325048 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.324462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37708)
0929 16:40:46.324709 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.324838 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:40:46.326099 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.325705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37718)
0929 16:40:46.325863 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.325949 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:40:46.326325 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
W20260929 16:40:46.356650 8291 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:46.356992 8291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:46.357062 8291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:46.357143 8291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:46.364416 8291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:46.364519 8291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:46.364576 8291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:46.364626 8291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:46.364686 8291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:46.374572 8291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:38185,127.17.205.61:35311,127.17.205.60:41641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.205.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:46.375842 8291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:46.377089 8291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:40:46.384174 8284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 4
I20260929 16:40:46.385358 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.384819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37734)
0929 16:40:46.385047 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.385158 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:40:46.386333 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.385968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37736)
0929 16:40:46.386112 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.386196 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:40:46.386554 8284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
W20260929 16:40:46.389230 8297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:46.389300 8296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:46.389328 8299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:46.390153 8291 server_base.cc:1034] running on GCE node
I20260929 16:40:46.390928 8291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:46.392416 8291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:46.393627 8291 hybrid_clock.cc:648] HybridClock initialized: now 1790700046393561 us; error 42 us; skew 500 ppm
I20260929 16:40:46.393941 8291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:46.396916 8291 init.cc:377] Logged in from keytab as kudu/127.17.205.60@KRBTEST.COM (short username kudu)
I20260929 16:40:46.398273 8291 webserver.cc:460] Webserver started at http://127.17.205.60:38909/ using document root <none> and password file <none>
I20260929 16:40:46.398952 8291 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:46.399055 8291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:46.399403 8291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:46.402271 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data/instance:
uuid: "194878d8fb6f45dca1cb16f3ae17c972"
format_stamp: "Formatted at 2026-09-29 16:40:46 on dist-test-slave-3h01"
server_key: "ac140a6e5764aa2c9b25ecb2f7510571"
server_key_iv: "2216e4204675cc826033cd9912ef4cb3"
server_key_version: "encryptionkey@0"
I20260929 16:40:46.403029 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal/instance:
uuid: "194878d8fb6f45dca1cb16f3ae17c972"
format_stamp: "Formatted at 2026-09-29 16:40:46 on dist-test-slave-3h01"
server_key: "ac140a6e5764aa2c9b25ecb2f7510571"
server_key_iv: "2216e4204675cc826033cd9912ef4cb3"
server_key_version: "encryptionkey@0"
I20260929 16:40:46.407938 8291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 16:40:46.411289 8306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:46.412569 8291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:40:46.412724 8291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "194878d8fb6f45dca1cb16f3ae17c972"
format_stamp: "Formatted at 2026-09-29 16:40:46 on dist-test-slave-3h01"
server_key: "ac140a6e5764aa2c9b25ecb2f7510571"
server_key_iv: "2216e4204675cc826033cd9912ef4cb3"
server_key_version: "encryptionkey@0"
I20260929 16:40:46.412912 8291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:46.432538 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 3
I20260929 16:40:46.432750 8291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:46.433672 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.433155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37738)
0929 16:40:46.433374 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.433477 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:40:46.434618 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.434274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37752)
0929 16:40:46.434442 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.434515 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:46.434811 8210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:41641: Network error: Client connection negotiation failed: client connection to 127.17.205.60:41641: connect: Connection refused (error 111)
I20260929 16:40:46.437178 8291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:46.437397 8291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:46.452912 8291 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.60:41641
I20260929 16:40:46.452934 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.60:41641 every 8 connection(s)
I20260929 16:40:46.454258 8291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
I20260929 16:40:46.455806 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8291
I20260929 16:40:46.455988 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699695913193-18228-0/minicluster-data/master-2/wal/instance
I20260929 16:40:46.456518 18228 external_mini_cluster.cc:1428] Setting key 863e20447d4e8006b10fc698dd7b2f5b
I20260929 16:40:46.458963 8358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:46.465978 8358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:46.479260 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.458429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.17.205.60:41641)
0929 16:40:46.459341 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.459353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.459401 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:40:46.460966 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.460989 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.461405 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.461804 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.463294 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.464603 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.465830 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.466543 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.466706 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.471172 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.471179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.471189 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.471262 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.474091 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.474881 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.474888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.474894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.475065 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.475480 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.475487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.475489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.476120 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.477194 (+ 1074us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.477485 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.477918 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":383,"threads_started":1}
I20260929 16:40:46.483294 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.468345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:38185 (local address 127.0.0.1:44298)
0929 16:40:46.469111 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.469176 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.469348 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.469738 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.469743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.470474 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.470795 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.470808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.471802 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.471807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.473076 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.473088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.473807 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.473814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.473985 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.474517 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.474539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.477023 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.480557 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.480568 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.480575 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.481062 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.481459 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.481464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.481467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.481633 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.482245 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.482250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.482395 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.482568 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.482867 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":340,"threads_started":1}
I20260929 16:40:46.484102 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.468438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.17.205.62:38185)
0929 16:40:46.468682 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.468690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.469284 (+ 594us) server_negotiation.cc:407] Connection header received
0929 16:40:46.469476 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.469482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.469566 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.469665 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.470958 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.471679 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.473213 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.473627 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.473746 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.477280 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.477288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.477293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.477334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.480427 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.481187 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.481193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.481197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.481347 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.481750 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.481754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.481757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.482099 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.482252 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.483589 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.483814 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:40:46.485268 8358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:46.498500 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.485955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:35311 (local address 127.0.0.1:49746)
0929 16:40:46.486241 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.486257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.486354 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.486640 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.486644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.487053 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.487256 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.487267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.488180 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.488185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.489306 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.489317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.490006 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.490012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.490182 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.490600 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.490622 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.492991 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.496140 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.496145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.496149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.496619 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.496952 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.496957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.496960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.497119 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.497589 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.497594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.497735 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.497870 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.498145 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 16:40:46.498529 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.486066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.17.205.61:35311)
0929 16:40:46.486243 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.486248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.486304 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:40:46.486463 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.486468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.486531 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.486612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.487402 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.488048 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.489436 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.489870 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.489975 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.493169 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.493176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.493180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.493222 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.496009 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.496720 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.496723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.496726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.496846 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.497207 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.497210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.497212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.497476 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.497607 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.497967 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.498167 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:40:46.499202 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.482683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.17.205.62:38185)
0929 16:40:46.483483 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.483488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.483511 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:40:46.483562 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.483566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.483630 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.485240 (+ 1610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.485523 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.486803 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.488271 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.488665 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.488750 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.493921 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.493927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.493930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.493963 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.496655 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.497368 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.497374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.497377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.497506 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.497843 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.497848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.497851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.498117 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.498361 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.498622 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.498832 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":448,"threads_started":1}
I20260929 16:40:46.499842 8358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:46.512409 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.500523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.17.205.60:41641)
0929 16:40:46.500684 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.500689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.500706 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:40:46.500787 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.500791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.500859 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.500953 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.501665 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.502342 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.503585 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.503972 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.504060 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.507169 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.507173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.507177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.507205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.509607 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.510358 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.510364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.510367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.510513 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.510887 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.510890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.510892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.511160 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.511362 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.511749 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.512135 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":62}
I20260929 16:40:46.512415 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.500435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37772)
0929 16:40:46.500586 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.500603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.500689 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.500991 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.500996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.501372 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.501528 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.501539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.502486 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.502491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.503457 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.503471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.504120 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.504126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.504310 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.504831 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.504852 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.507011 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.509718 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.509723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.509727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.510246 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.510622 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.510627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.510630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.510780 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.511331 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.511336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.511497 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.511673 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.512006 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 16:40:46.515198 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.501124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.17.205.61:35311)
0929 16:40:46.501272 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.501276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.501289 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:40:46.501324 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.501327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.501376 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.501444 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.502642 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.503384 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.504800 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.505186 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.505273 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.509667 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.509673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.509677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.509717 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.512863 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.513491 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.513497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.513499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.513649 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.514009 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.514012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.514013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.514288 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.514484 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.514784 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.514950 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:40:46.517825 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: Bootstrap starting.
I20260929 16:40:46.519811 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:46.521238 8358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:46.522265 8358 log.cc:824] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:46.525472 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: No bootstrap required, opened a new log
I20260929 16:40:46.522541 8185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7055ecb98f5a479c8b79bb693790173a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44304
I20260929 16:40:46.536175 8358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.536531 8358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194878d8fb6f45dca1cb16f3ae17c972, State: Initialized, Role: FOLLOWER
I20260929 16:40:46.537060 8358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.537990 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.538291 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.539120 8358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:40:46.548130 8376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:46.548265 8376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:46.546030 8291 master_runner.cc:186] Error getting master registration for 127.17.205.62:38185: OK, instance_id { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" instance_seqno: 1790700045472116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:40:46.553925 8291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:40:46.612021 8210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 4
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
I20260929 16:40:46.628458 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.612956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.17.205.60:41641)
0929 16:40:46.613168 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.613174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.613203 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:40:46.613279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.613285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.613367 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.613478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.614455 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.615628 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.617462 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.618175 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.618314 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.622413 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.622420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.622426 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.622481 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.625407 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.626253 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.626258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.626262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.626419 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.626814 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.626819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.626822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.627245 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.627401 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.627867 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.628101 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:40:46.628469 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.612775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37784)
0929 16:40:46.612999 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.613020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.613157 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.613516 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.613521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.614056 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.614275 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.614287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.615755 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.615760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.617292 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.617310 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.618332 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.618338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.618547 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.619198 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.619249 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.622151 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.625569 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.625575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.625580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.626130 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.626529 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.626535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.626539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.626709 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.627369 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.627374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.627587 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.627737 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.628046 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:40:46.634915 8210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: Bootstrap starting.
I20260929 16:40:46.639158 8210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:46.640374 8210 log.cc:824] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:46.643561 8210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: No bootstrap required, opened a new log
I20260929 16:40:46.648772 8210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.649148 8210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7055ecb98f5a479c8b79bb693790173a, State: Initialized, Role: FOLLOWER
I20260929 16:40:46.649647 8210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.650600 8379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.651006 8379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.651979 8210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:46.652047 8185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:40:46.652225 8185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:46.655910 8185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.657822 8185 leader_election.cc:288] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [CANDIDATE]: Term 1 election: Requested vote from peers 8a9e7ba61771426cbceed38babf9157e (127.17.205.61:35311), 194878d8fb6f45dca1cb16f3ae17c972 (127.17.205.60:41641)
I20260929 16:40:46.660012 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7055ecb98f5a479c8b79bb693790173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "194878d8fb6f45dca1cb16f3ae17c972"
I20260929 16:40:46.660543 8333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:46.664129 8333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7055ecb98f5a479c8b79bb693790173a in term 1.
I20260929 16:40:46.664887 8161 leader_election.cc:302] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194878d8fb6f45dca1cb16f3ae17c972, 7055ecb98f5a479c8b79bb693790173a; no voters:
I20260929 16:40:46.665510 8379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:46.675575 8379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [term 1 LEADER]: Becoming Leader. State: Replica: 7055ecb98f5a479c8b79bb693790173a, State: Running, Role: LEADER
I20260929 16:40:46.676725 8379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.685750 8284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } attempt: 5
I20260929 16:40:46.688599 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7055ecb98f5a479c8b79bb693790173a. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.689079 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: This master's current role is: LEADER
W20260929 16:40:46.689308 8312 tablet.cc:2369] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:46.691136 8396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 16:40:46 dist-test-slave-3h01 krb5kdc[8130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700045, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
I20260929 16:40:46.706684 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.686526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:41641 (local address 127.0.0.1:37792)
0929 16:40:46.686967 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:46.686987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:46.687131 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:46.688133 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:46.688138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:46.688647 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:46.688869 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.688882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.690300 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.690306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.691615 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:46.691631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.692759 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.692765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:46.693033 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.693747 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:46.693791 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:46.697886 (+ 4095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:46.702345 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.702352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.702357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.702970 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.704337 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:46.704345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:46.704349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:46.704532 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:46.705590 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:46.705596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:46.705785 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 16:40:46.705935 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:40:46.706265 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
I20260929 16:40:46.708570 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.687377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.17.205.60:41641)
0929 16:40:46.687550 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.687555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.687575 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:40:46.687737 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.687743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.687838 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.687930 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.689063 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.690138 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.691784 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.692428 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.692555 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.698200 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.698208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.698213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.698262 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.702114 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.703155 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.703161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.703165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.703359 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.704727 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.704755 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.704761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.705387 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.707190 (+ 1803us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.707476 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.707925 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 16:40:46.712354 8164 tablet.cc:2369] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:46.714812 8284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: Bootstrap starting.
I20260929 16:40:46.704598 8143 master_runner.cc:186] Error getting master registration for 127.17.205.61:35311: OK, instance_id { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" instance_seqno: 1790700045963159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:40:46.715636 8143 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:40:46.719750 8284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:46.721182 8284 log.cc:824] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:46.725106 8284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: No bootstrap required, opened a new log
I20260929 16:40:46.736634 8284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.737116 8284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9e7ba61771426cbceed38babf9157e, State: Initialized, Role: FOLLOWER
I20260929 16:40:46.737689 8284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } }
I20260929 16:40:46.739022 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.739830 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.740350 8284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:46.740398 8259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7055ecb98f5a479c8b79bb693790173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8a9e7ba61771426cbceed38babf9157e"
I20260929 16:40:46.740916 8259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:46.744571 8259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7055ecb98f5a479c8b79bb693790173a in term 1.
I20260929 16:40:46.754880 8396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:46.757053 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.722629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.17.205.60:41641)
0929 16:40:46.722875 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.722881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.733259 (+ 10378us) server_negotiation.cc:407] Connection header received
0929 16:40:46.733405 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.733411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.733512 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.733607 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.742579 (+ 8972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.743821 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.745812 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.746514 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.746635 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.749918 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.749924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.749929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.749974 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.753654 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.754571 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.754576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.754580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.754745 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.755193 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.755198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.755202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.755588 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.755821 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.756377 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.756646 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 16:40:46.775701 8414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:46.775852 8414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:46.776842 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.695434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.17.205.61:35311)
0929 16:40:46.695793 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.695798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.725005 (+ 29207us) server_negotiation.cc:407] Connection header received
0929 16:40:46.725158 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.725164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.725245 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.725338 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.730380 (+ 5042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.731597 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.739642 (+ 8045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.740341 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.740456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.767785 (+ 27329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.767792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.767797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.767839 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.771405 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.772278 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.772283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.772287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.772467 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.773634 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.773640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.773642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.774046 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.774279 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.776190 (+ 1911us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.776467 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260929 16:40:46.777449 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:46.695249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.17.205.62:38185)
0929 16:40:46.695654 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:46.695660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:46.722001 (+ 26341us) server_negotiation.cc:407] Connection header received
0929 16:40:46.724121 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:46.724127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:46.724223 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:46.724336 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:46.727449 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.728553 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.735277 (+ 6724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:46.736000 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:46.736126 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:46.763184 (+ 27058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:46.763193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:46.763199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:46.763273 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:46.766739 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.772977 (+ 6238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.772983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.772988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.773163 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:46.774775 (+ 1612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:46.774781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:46.774785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:46.775155 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:46.775452 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:46.776700 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0929 16:40:46.777117 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260929 16:40:46.792349 8259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [term 1 FOLLOWER]: Refusing update from remote peer 7055ecb98f5a479c8b79bb693790173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:46.794186 8388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:46.796376 8333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [term 1 FOLLOWER]: Refusing update from remote peer 7055ecb98f5a479c8b79bb693790173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:46.798211 8379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:46.837224 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7055ecb98f5a479c8b79bb693790173a. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.837576 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.843716 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7055ecb98f5a479c8b79bb693790173a. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.843950 8416 mvcc.cc:205] Tried to move back new op lower bound from 7334707391650816000 to 7334707391197765632. Current Snapshot: MvccSnapshot[applied={T|T < 7334707391650816000}]
I20260929 16:40:46.844029 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.849004 8418 mvcc.cc:205] Tried to move back new op lower bound from 7334707391650816000 to 7334707391197765632. Current Snapshot: MvccSnapshot[applied={T|T < 7334707391650816000}]
I20260929 16:40:46.854055 8417 mvcc.cc:205] Tried to move back new op lower bound from 7334707391650816000 to 7334707391197765632. Current Snapshot: MvccSnapshot[applied={T|T < 7334707391650816000}]
I20260929 16:40:46.859261 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.859620 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:46.860064 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.860312 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:46.863083 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.863380 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.864217 8396 catalog_manager.cc:1347] Generated new cluster ID: f16ecf8001354fdb9a40197b7f8476bb
I20260929 16:40:46.864322 8396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:46.872747 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7055ecb98f5a479c8b79bb693790173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7055ecb98f5a479c8b79bb693790173a" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 38185 } } peers { permanent_uuid: "8a9e7ba61771426cbceed38babf9157e" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 35311 } } peers { permanent_uuid: "194878d8fb6f45dca1cb16f3ae17c972" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 41641 } } }
I20260929 16:40:46.873018 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:46.891335 8396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:46.893301 8396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:46.915421 8396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7055ecb98f5a479c8b79bb693790173a: Generated new TSK 0
I20260929 16:40:46.916596 8396 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:40:47.056998 8363 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.043320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.17.205.62:38185)
0929 16:40:47.043510 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.043517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.054702 (+ 11185us) server_negotiation.cc:407] Connection header received
0929 16:40:47.055353 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.055362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.055464 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.055540 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.056702 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44316: BlockingRecv error: recv got EOF from 127.0.0.1:44316 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:40:47.059712 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.050212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.17.205.61:35311)
0929 16:40:47.050435 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.050443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.058051 (+ 7608us) server_negotiation.cc:407] Connection header received
0929 16:40:47.058347 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.058354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.058453 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.058530 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.059484 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49774: BlockingRecv error: recv got EOF from 127.0.0.1:49774 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 16:40:47.068648 8361 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.066229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.17.205.60:41641)
0929 16:40:47.066438 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.066445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.066928 (+ 483us) server_negotiation.cc:407] Connection header received
0929 16:40:47.067183 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.067188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.067289 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.067370 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.068411 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37808: BlockingRecv error: recv got EOF from 127.0.0.1:37808 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:40:47.090929 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.088615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.17.205.61:35311)
0929 16:40:47.088792 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.088797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.089325 (+ 528us) server_negotiation.cc:407] Connection header received
0929 16:40:47.089545 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.089552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.089684 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.089802 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.090592 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49776: BlockingRecv error: recv got EOF from 127.0.0.1:49776 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:40:47.097820 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.096337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.17.205.61:35311)
0929 16:40:47.096497 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.096502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.096519 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:40:47.096586 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.096592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.096675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.096786 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.097602 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49786: BlockingRecv error: recv got EOF from 127.0.0.1:49786 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 16:40:47.107340 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.105081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.17.205.61:35311)
0929 16:40:47.105265 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.105271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.105811 (+ 540us) server_negotiation.cc:407] Connection header received
0929 16:40:47.106008 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.106014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.106121 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.106227 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.107037 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49800: BlockingRecv error: recv got EOF from 127.0.0.1:49800 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:40:47.117131 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.114721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.17.205.61:35311)
0929 16:40:47.114958 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.114963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.115544 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:40:47.115742 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.115748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.115847 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.115985 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.116874 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49808: BlockingRecv error: recv got EOF from 127.0.0.1:49808 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260929 16:40:47.135651 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.133380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.17.205.61:35311)
0929 16:40:47.133569 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.133576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.134087 (+ 511us) server_negotiation.cc:407] Connection header received
0929 16:40:47.134327 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.134337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.134438 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.134547 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.135415 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49816: BlockingRecv error: recv got EOF from 127.0.0.1:49816 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 16:40:47.158066 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.155735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.17.205.61:35311)
0929 16:40:47.155934 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.155941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.156493 (+ 552us) server_negotiation.cc:407] Connection header received
0929 16:40:47.156701 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.156707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.156825 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.156937 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.157811 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49832: BlockingRecv error: recv got EOF from 127.0.0.1:49832 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:40:47.207644 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.205137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.17.205.61:35311)
0929 16:40:47.205359 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.205365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.205926 (+ 561us) server_negotiation.cc:407] Connection header received
0929 16:40:47.206148 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.206154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.206258 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.206401 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.207361 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49844: BlockingRecv error: recv got EOF from 127.0.0.1:49844 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 16:40:47.279057 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.276967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.17.205.61:35311)
0929 16:40:47.277164 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.277175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.277651 (+ 476us) server_negotiation.cc:407] Connection header received
0929 16:40:47.277851 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.277856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.277956 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.278043 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.278804 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49846: BlockingRecv error: recv got EOF from 127.0.0.1:49846 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:40:47.412467 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.410489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.17.205.61:35311)
0929 16:40:47.410725 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.410731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.410993 (+ 262us) server_negotiation.cc:407] Connection header received
0929 16:40:47.411242 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.411248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.411338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.411438 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.412234 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49852: BlockingRecv error: recv got EOF from 127.0.0.1:49852 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:40:47.550164 8376 catalog_manager.cc:1259] Loaded cluster ID: f16ecf8001354fdb9a40197b7f8476bb
I20260929 16:40:47.550273 8376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: loading cluster ID for follower catalog manager: success
I20260929 16:40:47.552835 8376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: acquiring CA information for follower catalog manager: success
I20260929 16:40:47.554593 8376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 194878d8fb6f45dca1cb16f3ae17c972: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:40:47.678625 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:47.676941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.17.205.61:35311)
0929 16:40:47.677149 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:47.677155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:47.677282 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:40:47.677504 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:47.677510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:47.677590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:47.677677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:47.678413 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49860: BlockingRecv error: recv got EOF from 127.0.0.1:49860 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:40:47.777838 8414 catalog_manager.cc:1259] Loaded cluster ID: f16ecf8001354fdb9a40197b7f8476bb
I20260929 16:40:47.777951 8414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: loading cluster ID for follower catalog manager: success
I20260929 16:40:47.780452 8414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: acquiring CA information for follower catalog manager: success
I20260929 16:40:47.782260 8414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a9e7ba61771426cbceed38babf9157e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:40:48.199927 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:48.194067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.17.205.61:35311)
0929 16:40:48.194551 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:48.194558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:48.194667 (+ 109us) server_negotiation.cc:407] Connection header received
0929 16:40:48.194873 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:48.194878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:48.195029 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:48.195140 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:48.195609 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.196322 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.197709 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.198114 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.198228 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:48.198723 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:48.199281 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:48.199419 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:48.199527 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:40:48.199616 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260929 16:40:48.211524 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:48.205922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.17.205.60:41641)
0929 16:40:48.206400 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:48.206407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:48.206474 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:40:48.206710 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:48.206714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:48.206864 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:48.207015 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:48.207453 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.208123 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.209393 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.209777 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.209886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:48.210385 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:48.210901 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:48.211047 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:48.211124 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:40:48.211234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":495,"threads_started":1}
I20260929 16:40:48.226691 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:48.218077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.17.205.61:35311)
0929 16:40:48.218296 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:48.218302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:48.220686 (+ 2384us) server_negotiation.cc:407] Connection header received
0929 16:40:48.220825 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:48.220830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:48.220921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:48.221011 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:48.221749 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.222750 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.224588 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.225185 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.225286 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:48.225897 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:48.226029 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:48.226109 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:48.226315 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 16:40:48.226404 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:40:48.227069 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:48.221229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.17.205.60:41641)
0929 16:40:48.221386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:48.221392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:48.221815 (+ 423us) server_negotiation.cc:407] Connection header received
0929 16:40:48.221959 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:48.221962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:48.222026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:48.222097 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:48.222569 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.223633 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.225358 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.225962 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.226064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:48.226457 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:48.226569 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:48.226634 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:48.226755 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:40:48.226828 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:40:48.227710 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:48.218088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.17.205.62:38185)
0929 16:40:48.219644 (+ 1556us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:48.219651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:48.220174 (+ 523us) server_negotiation.cc:407] Connection header received
0929 16:40:48.220400 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:48.220406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:48.220629 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:48.220831 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:48.221358 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.222475 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.224467 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:48.225078 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:48.225203 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:48.225731 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:48.226189 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:48.226314 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:48.227361 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 16:40:48.227448 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":1311,"threads_started":1}
I20260929 16:40:48.243257 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8143
I20260929 16:40:48.244408 8205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:48.356003 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8143
I20260929 16:40:48.373715 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8217
I20260929 16:40:48.374779 8279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:48.481088 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8217
I20260929 16:40:48.498340 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8291
I20260929 16:40:48.499418 8353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:48.606298 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8291
2026-09-29T16:40:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4828 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:40:48 dist-test-slave-3h01 krb5kdc[8508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:40:48 dist-test-slave-3h01 krb5kdc[8508](info): set up 2 sockets
Sep 29 16:40:48 dist-test-slave-3h01 krb5kdc[8508](info): commencing operation
krb5kdc: starting...
W20260929 16:40:49.623764 18228 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/test-user.keytab.
Sep 29 16:40:49 dist-test-slave-3h01 krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:40:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.205.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
Entry for principal kudu/127.17.205.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab.
I20260929 16:40:49.873818 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45021
--webserver_interface=127.17.205.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:40:50.195500 8521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:50.195883 8521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:50.195991 8521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:50.196096 8521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:50.203480 8521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:50.203570 8521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:50.203640 8521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:50.203716 8521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:50.203799 8521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:50.214025 8521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.62.keytab
--principal=kudu/127.17.205.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.62:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.205.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:50.215386 8521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:50.216701 8521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:40:50.228211 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:50.228116 8527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:50.228132 8529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:50.229123 8521 server_base.cc:1034] running on GCE node
I20260929 16:40:50.229895 8521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:50.231374 8521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:50.232582 8521 hybrid_clock.cc:648] HybridClock initialized: now 1790700050232507 us; error 51 us; skew 500 ppm
I20260929 16:40:50.232874 8521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:50 dist-test-slave-3h01 krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:50.236357 8521 init.cc:377] Logged in from keytab as kudu/127.17.205.62@KRBTEST.COM (short username kudu)
I20260929 16:40:50.237785 8521 webserver.cc:460] Webserver started at http://127.17.205.62:46165/ using document root <none> and password file <none>
I20260929 16:40:50.238492 8521 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:50.238583 8521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:50.238891 8521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:50.241901 8521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data/instance:
uuid: "9c55e4b4f6f84ddc86212ba172578664"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f8d6ab7760adea9aaa29c528ea496dea"
server_key_iv: "42a0641e03cca06c38b97b246964e29b"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.242691 8521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance:
uuid: "9c55e4b4f6f84ddc86212ba172578664"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f8d6ab7760adea9aaa29c528ea496dea"
server_key_iv: "42a0641e03cca06c38b97b246964e29b"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.247828 8521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:40:50.251291 8536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:50.252624 8521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:50.252785 8521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal
uuid: "9c55e4b4f6f84ddc86212ba172578664"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f8d6ab7760adea9aaa29c528ea496dea"
server_key_iv: "42a0641e03cca06c38b97b246964e29b"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.253005 8521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:50.269924 8521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:50.273842 8521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:50.274116 8521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:50.291021 8521 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.62:45021
I20260929 16:40:50.291042 8587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.62:45021 every 8 connection(s)
I20260929 16:40:50.292429 8521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/data/info.pb
I20260929 16:40:50.295572 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8521
I20260929 16:40:50.295754 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-0/wal/instance
I20260929 16:40:50.296296 18228 external_mini_cluster.cc:1428] Setting key d2fc815d4a87c0b08003ef02c06347c0
I20260929 16:40:50.296578 8588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:50.305488 8588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:50 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.205.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.61@KRBTEST.COM" created.
I20260929 16:40:50.330782 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.310210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.17.205.62:45021)
0929 16:40:50.310802 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:50.310810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:50.310856 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:40:50.312714 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:50.312720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:50.312982 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:50.313147 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:50.314921 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.316171 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.318022 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.318577 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.318706 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.322951 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:50.322963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:50.322983 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:50.323028 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:50.326554 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.327376 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.327384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.327386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.327561 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.327966 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.327972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.327976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.328552 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:50.328723 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:50.329356 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:40:50.329679 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":434,"thread_start_us":285,"threads_started":1}
I20260929 16:40:50.330807 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.309094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45021 (local address 127.0.0.1:37796)
0929 16:40:50.309929 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.309982 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:50.312317 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:50.313214 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:50.313221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:50.314220 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:50.314690 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.314714 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.316331 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.316337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.317886 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.317905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.318815 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.318822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.319010 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.319740 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:50.319777 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:50.322649 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:50.326699 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.326711 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.326721 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.327250 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.327664 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.327669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.327673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.327826 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.328699 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:50.328708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:50.328953 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 16:40:50.329226 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 16:40:50.329838 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":95,"thread_start_us":310,"threads_started":1}
I20260929 16:40:50.334048 8588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:50.335081 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.334701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42822)
0929 16:40:50.334857 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.334965 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:40:50.335351 8539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
I20260929 16:40:50.336839 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.336487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42830)
0929 16:40:50.336636 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.336705 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:50.337033 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:34527: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
Entry for principal kudu/127.17.205.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab.
I20260929 16:40:50.357743 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:34527
--webserver_interface=127.17.205.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:40:50.386349 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } attempt: 1
I20260929 16:40:50.387503 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.387046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42840)
0929 16:40:50.387253 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.387367 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:40:50.388437 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.388101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42846)
0929 16:40:50.388244 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.388309 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:40:50.388618 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:34527: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
I20260929 16:40:50.456832 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } attempt: 2
I20260929 16:40:50.458042 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.457552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42852)
0929 16:40:50.457791 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.457896 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:40:50.459036 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.458683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42864)
0929 16:40:50.458827 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.458913 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:50.459246 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:34527: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
I20260929 16:40:50.550470 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } attempt: 3
I20260929 16:40:50.551643 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.551183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42874)
0929 16:40:50.551421 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.551511 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 16:40:50.552634 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.552280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42890)
0929 16:40:50.552428 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.552498 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:40:50.552814 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:34527: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
W20260929 16:40:50.673559 8595 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:50.673898 8595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:50.673995 8595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:50.674108 8595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:50.681524 8595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:50.681623 8595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:50.681703 8595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:50.681771 8595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:50.681854 8595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:50.691573 8595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.61.keytab
--principal=kudu/127.17.205.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.61:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.205.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:50.692888 8595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:50.694202 8595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:40:50.696033 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } attempt: 4
I20260929 16:40:50.697082 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.696586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42904)
0929 16:40:50.696859 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.696950 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 16:40:50.698072 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.697708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42920)
0929 16:40:50.697845 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.697955 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:40:50.698249 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.61:34527: Network error: Client connection negotiation failed: client connection to 127.17.205.61:34527: connect: Connection refused (error 111)
W20260929 16:40:50.705497 8600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:50.705439 8601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:50.705786 8603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:50.706370 8595 server_base.cc:1034] running on GCE node
I20260929 16:40:50.707069 8595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:50.708477 8595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:50.709707 8595 hybrid_clock.cc:648] HybridClock initialized: now 1790700050709647 us; error 33 us; skew 500 ppm
I20260929 16:40:50.709995 8595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:50 dist-test-slave-3h01 krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:50.712970 8595 init.cc:377] Logged in from keytab as kudu/127.17.205.61@KRBTEST.COM (short username kudu)
I20260929 16:40:50.714357 8595 webserver.cc:460] Webserver started at http://127.17.205.61:46185/ using document root <none> and password file <none>
I20260929 16:40:50.714993 8595 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:50.715075 8595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:50.715397 8595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:50.718303 8595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data/instance:
uuid: "e8c0f96111b1423ea3d243e6be17f186"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f89c9be491d4972dc6d8cac7a23ae2ba"
server_key_iv: "460e63259dc5f3546c4e7e5096f8956f"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.719053 8595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal/instance:
uuid: "e8c0f96111b1423ea3d243e6be17f186"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f89c9be491d4972dc6d8cac7a23ae2ba"
server_key_iv: "460e63259dc5f3546c4e7e5096f8956f"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.724063 8595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:40:50.727246 8610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:50.728518 8595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:40:50.728688 8595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal
uuid: "e8c0f96111b1423ea3d243e6be17f186"
format_stamp: "Formatted at 2026-09-29 16:40:50 on dist-test-slave-3h01"
server_key: "f89c9be491d4972dc6d8cac7a23ae2ba"
server_key_iv: "460e63259dc5f3546c4e7e5096f8956f"
server_key_version: "encryptionkey@0"
I20260929 16:40:50.728873 8595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:50.745220 8595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:50.749143 8595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:50.749378 8595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:50.765777 8595 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.61:34527
I20260929 16:40:50.765797 8661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.61:34527 every 8 connection(s)
I20260929 16:40:50.767027 8595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/data/info.pb
I20260929 16:40:50.769285 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8595
I20260929 16:40:50.769444 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-1/wal/instance
I20260929 16:40:50.769876 18228 external_mini_cluster.cc:1428] Setting key d2b6b1cebbfebd07ecf2e0ed8810c890
I20260929 16:40:50.770766 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:50.777308 8662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.205.60@KRBTEST.COM; defaulting to no policy
Sep 29 16:40:50 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.205.60@KRBTEST.COM" created.
I20260929 16:40:50.797416 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.780057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.17.205.62:45021)
0929 16:40:50.780266 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:50.780272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:50.781388 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 16:40:50.783346 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:50.783351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:50.783435 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:50.783527 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:50.785000 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.785988 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.787535 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.788195 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.788335 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.791831 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:50.791837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:50.791840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:50.791877 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:50.794387 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.795101 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.795105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.795108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.795260 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.795668 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.795673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.795677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.795980 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:50.796112 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:50.796851 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 16:40:50.797096 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:40:50.798382 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.780441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45021 (local address 127.0.0.1:37806)
0929 16:40:50.781221 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.781274 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:50.783035 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:50.783718 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:50.783726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:50.784273 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:50.784828 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.784848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.786121 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.786124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.787375 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.787387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.788388 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.788393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.788598 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.789247 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:50.789270 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:50.791549 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:50.794521 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.794531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.794546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.794975 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.795361 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.795374 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.795376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.795553 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.796100 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:50.796108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:50.796485 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 16:40:50.796718 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 16:40:50.797231 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":309,"threads_started":1}
I20260929 16:40:50.800081 8662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:50 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:50.814512 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.800694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42926)
0929 16:40:50.800924 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.800938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:50.801019 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:50.801967 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:50.801972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:50.802390 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:50.802526 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.802535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.803875 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.803880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.804704 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.804715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.805487 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.805493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:50.805649 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.806184 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:50.806201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:50.808349 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:50.812220 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.812224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.812227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.812582 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.812983 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:50.812986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:50.812989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:50.813116 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:50.813816 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:50.813820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:50.813943 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 16:40:50.814047 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:40:50.814249 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 16:40:50.814919 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.800972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.17.205.61:34527)
0929 16:40:50.801572 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:50.801578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:50.801601 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:40:50.801646 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:50.801651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:50.801825 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:50.801966 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:50.802678 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.803755 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.804824 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:50.805356 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:50.805465 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:50.808583 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:50.808601 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:50.808620 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:50.808656 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:50.812082 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.812723 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.812732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.812736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.812877 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:50.813219 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:50.813224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:50.813227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:50.813694 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:50.813919 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:50.814227 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:40:50.814491 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":336,"threads_started":1}
I20260929 16:40:50.817313 8662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:50.818442 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.818028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:52970)
0929 16:40:50.818186 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.818302 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:40:50.818696 8612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:50.820108 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.819794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:52980)
0929 16:40:50.819936 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.820012 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 16:40:50.820343 8662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
Entry for principal kudu/127.17.205.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab.
I20260929 16:40:50.824060 18228 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
/tmp/dist-test-taskaaWXO3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:34237
--webserver_interface=127.17.205.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.205.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:40:50.869674 8662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 1
I20260929 16:40:50.870826 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.870394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:52984)
0929 16:40:50.870572 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.870681 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:40:50.871824 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.871509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:52994)
0929 16:40:50.871647 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.871722 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:40:50.872048 8662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:50.940243 8662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 2
I20260929 16:40:50.941334 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.940913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53004)
0929 16:40:50.941091 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.941183 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:40:50.942242 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.941926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53014)
0929 16:40:50.942051 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.942132 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:40:50.942453 8662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:50.997478 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } attempt: 5
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:51.011039 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.998182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42928)
0929 16:40:50.998386 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:50.998404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:50.998511 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:50.998960 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:50.998966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:50.999437 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:50.999620 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:50.999630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.000608 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.000613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.001489 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.001503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.002225 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.002231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.002428 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.003019 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.003038 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.005692 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.008641 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.008647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.008652 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.009059 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.009431 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.009436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.009440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.009582 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.010095 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.010099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.010257 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.010391 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.010685 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:40:51.011041 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:50.998463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.17.205.61:34527)
0929 16:40:50.998664 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:50.998668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:50.998683 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:40:50.998733 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:50.998736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:50.998810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:50.998901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:50.999771 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.000472 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.001635 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.002079 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.002187 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.005927 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.005933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.005936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.005970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.008457 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.009183 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.009188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.009190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.009314 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.009684 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.009689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.009690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.009983 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.010116 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.010498 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.010707 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:40:51.012089 8588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:40:51.013403 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.012831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53030)
0929 16:40:51.013058 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.013175 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:40:51.014495 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.014034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53034)
0929 16:40:51.014229 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.014346 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:40:51.014760 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:51.033641 8662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 3
I20260929 16:40:51.034837 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.034234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53048)
0929 16:40:51.034489 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.034624 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:40:51.036037 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.035492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53056)
0929 16:40:51.035752 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.035867 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 16:40:51.036278 8662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:51.066031 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 1
I20260929 16:40:51.067365 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.066740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53060)
0929 16:40:51.066971 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.067097 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:40:51.068478 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.068027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53076)
0929 16:40:51.068227 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.068330 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:40:51.068719 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:51.136965 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 2
I20260929 16:40:51.138253 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.137660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53092)
0929 16:40:51.137907 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.138022 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:40:51.139374 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.138917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53102)
0929 16:40:51.139090 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.139229 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:40:51.139585 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
W20260929 16:40:51.159821 8669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:40:51.160158 8669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:40:51.160238 8669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:40:51.160329 8669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:40:51.167661 8669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:40:51.167773 8669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:40:51.167836 8669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:40:51.167893 8669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:40:51.167961 8669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:40:51.177860 8669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.205.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.205.62:45021,127.17.205.61:34527,127.17.205.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/krb5kdc/kudu_127.17.205.60.keytab
--principal=kudu/127.17.205.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.205.60:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.205.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:40:51.179176 8669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:40:51.179478 8662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 4
I20260929 16:40:51.180663 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.180056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53106)
0929 16:40:51.180342 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.180459 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:40:51.180809 8669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:40:51.181711 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.181332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53122)
0929 16:40:51.181487 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.181579 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:40:51.181927 8662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
W20260929 16:40:51.192620 8675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:51.192655 8677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:40:51.192699 8674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:40:51.193498 8669 server_base.cc:1034] running on GCE node
I20260929 16:40:51.194247 8669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:40:51.195683 8669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:40:51.196898 8669 hybrid_clock.cc:648] HybridClock initialized: now 1790700051196828 us; error 39 us; skew 500 ppm
I20260929 16:40:51.197212 8669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700051, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:40:51.200244 8669 init.cc:377] Logged in from keytab as kudu/127.17.205.60@KRBTEST.COM (short username kudu)
I20260929 16:40:51.201731 8669 webserver.cc:460] Webserver started at http://127.17.205.60:37969/ using document root <none> and password file <none>
I20260929 16:40:51.202406 8669 fs_manager.cc:359] Metadata directory not provided
I20260929 16:40:51.202518 8669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:40:51.202817 8669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:40:51.205781 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data/instance:
uuid: "6e7a80bbc808499981a71e1d6590902f"
format_stamp: "Formatted at 2026-09-29 16:40:51 on dist-test-slave-3h01"
server_key: "ad291188db6568babc8237143561a425"
server_key_iv: "e1c5e493e33afc949e18eef3704a6e0f"
server_key_version: "encryptionkey@0"
I20260929 16:40:51.206585 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal/instance:
uuid: "6e7a80bbc808499981a71e1d6590902f"
format_stamp: "Formatted at 2026-09-29 16:40:51 on dist-test-slave-3h01"
server_key: "ad291188db6568babc8237143561a425"
server_key_iv: "e1c5e493e33afc949e18eef3704a6e0f"
server_key_version: "encryptionkey@0"
I20260929 16:40:51.211828 8669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:40:51.215200 8684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:40:51.216575 8669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:40:51.216747 8669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data,/tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal
uuid: "6e7a80bbc808499981a71e1d6590902f"
format_stamp: "Formatted at 2026-09-29 16:40:51 on dist-test-slave-3h01"
server_key: "ad291188db6568babc8237143561a425"
server_key_iv: "e1c5e493e33afc949e18eef3704a6e0f"
server_key_version: "encryptionkey@0"
I20260929 16:40:51.216933 8669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:40:51.241748 8669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:40:51.245801 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 3
I20260929 16:40:51.246023 8669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:40:51.246304 8669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:40:51.246901 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.246432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53136)
0929 16:40:51.246664 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.246746 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:40:51.247963 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.247588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53148)
0929 16:40:51.247781 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.247859 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 16:40:51.248210 8588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.205.60:34237: Network error: Client connection negotiation failed: client connection to 127.17.205.60:34237: connect: Connection refused (error 111)
I20260929 16:40:51.263538 8669 rpc_server.cc:307] RPC server started. Bound to: 127.17.205.60:34237
I20260929 16:40:51.263549 8735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.205.60:34237 every 8 connection(s)
I20260929 16:40:51.264851 8669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/data/info.pb
I20260929 16:40:51.267699 18228 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu as pid 8669
I20260929 16:40:51.267870 18228 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaaWXO3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699695913193-18228-0/minicluster-data/master-2/wal/instance
I20260929 16:40:51.268254 18228 external_mini_cluster.cc:1428] Setting key 87033ba2f14f429096a81d3e1f4b8e0f
I20260929 16:40:51.268661 8736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:40:51.278426 8736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700051, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
I20260929 16:40:51.290488 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.269890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.17.205.60:34237)
0929 16:40:51.270628 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.270642 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.270691 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:40:51.272341 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.272368 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.272902 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.273357 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.274566 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.276568 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.277819 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.278912 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.279095 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.283359 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.283367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.283377 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.283411 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.286333 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.286966 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.286974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.286982 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.287194 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.287638 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.287643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.287646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.288195 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.288424 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.288909 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.289341 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":298,"threads_started":1}
I20260929 16:40:51.296494 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.281505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.17.205.62:45021)
0929 16:40:51.281718 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.281723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.282360 (+ 637us) server_negotiation.cc:407] Connection header received
0929 16:40:51.282653 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.282659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.282743 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.282850 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.283991 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.285012 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.286403 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.286859 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.286973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.290306 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.290313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.290318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.290358 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.293503 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.294299 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.294306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.294309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.294483 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.294902 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.294908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.294913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.295305 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.295442 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.295942 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.296156 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:40:51.296535 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.281481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.62:45021 (local address 127.0.0.1:37814)
0929 16:40:51.282230 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.282271 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:51.282500 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:51.282901 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:51.282906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:51.283568 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:51.283840 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.283849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.285168 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.285172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.286268 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.286279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.287037 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.287041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.287170 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.287788 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.287805 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.290090 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.293620 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.293628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.293635 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.294151 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.294631 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.294635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.294639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.294784 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.295428 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.295435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.295628 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.295820 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.296115 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":375,"threads_started":1}
I20260929 16:40:51.298444 8736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.62@KRBTEST.COM
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700051, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:51.309320 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.294064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.17.205.62:45021)
0929 16:40:51.295346 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.295351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.295380 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:40:51.295424 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.295429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.295495 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.295587 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.297007 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.298089 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.299766 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.300203 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.300307 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.304253 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.304258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.304261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.304291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.306833 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.307488 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.307493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.307496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.307609 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.307956 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.307959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.307961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.308218 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.308459 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.308767 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.308989 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":920,"threads_started":1}
I20260929 16:40:51.310443 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.299086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.61:34527 (local address 127.0.0.1:42944)
0929 16:40:51.299265 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.299283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:51.299375 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:51.299685 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:51.299690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:51.300101 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:51.300270 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.300280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.301379 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.301384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.302241 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.302251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.302933 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.302939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.303092 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.303583 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.303607 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.305574 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.308268 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.308273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.308277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.308610 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.308949 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.308954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.308958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.309084 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.309643 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.309646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.309773 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.309925 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.310204 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:40:51.310555 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.299201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.17.205.61:34527)
0929 16:40:51.299387 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.299393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.299435 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:40:51.299491 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.299496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.299572 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.299664 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.300428 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.301248 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.302379 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.302792 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.302897 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.305741 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.305746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.305749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.305781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.308150 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.308716 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.308719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.308722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.308848 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.309206 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.309211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.309214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.309520 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.309668 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.310004 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.310214 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:40:51.311316 8736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700051, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.60@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.205.61@KRBTEST.COM
I20260929 16:40:51.325101 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.312059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53156)
0929 16:40:51.312189 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.312206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:51.312305 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:51.312558 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:51.312564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:51.312991 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:51.313149 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.313160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.314404 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.314409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.315780 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.315794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.316678 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.316683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.316905 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.317435 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.317455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.319801 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.322881 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.322886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.322891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.323246 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.323570 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.323574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.323578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.323700 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.324161 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.324165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.324299 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.324415 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.324693 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 16:40:51.325182 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.312006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.17.205.60:34237)
0929 16:40:51.312168 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.312173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.312253 (+ 80us) server_negotiation.cc:407] Connection header received
0929 16:40:51.312393 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.312397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.312445 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.312511 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.313269 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.314282 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.315919 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.316533 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.316650 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.319959 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.319964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.319969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.319995 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.322759 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.323332 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.323337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.323340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.323466 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.323799 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.323804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.323808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.324059 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.324181 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.324522 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.324858 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":54}
I20260929 16:40:51.326203 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.311128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.17.205.61:34527)
0929 16:40:51.311299 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.311304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.311463 (+ 159us) server_negotiation.cc:407] Connection header received
0929 16:40:51.311597 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.311601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.311653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.311731 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.313199 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.314282 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.315898 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.316536 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.316651 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.321113 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.321118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.321121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.321150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.323649 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.324254 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.324258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.324260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.324399 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.324739 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.324742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.324744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.324989 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.325631 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.325771 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.325970 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:40:51.331022 8736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: Bootstrap starting.
I20260929 16:40:51.331967 18228 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:40:51.334352 8736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:51.335444 8736 log.cc:824] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:51.334373 8563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9c55e4b4f6f84ddc86212ba172578664"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37824
I20260929 16:40:51.338203 8736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: No bootstrap required, opened a new log
I20260929 16:40:51.345230 8736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.345592 8736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7a80bbc808499981a71e1d6590902f, State: Initialized, Role: FOLLOWER
I20260929 16:40:51.346035 8736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.346867 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.347172 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.348237 8736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:40:51.357563 8754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:51.357702 8754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:51.355391 8669 master_runner.cc:186] Error getting master registration for 127.17.205.62:45021: OK, instance_id { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" instance_seqno: 1790700050286062 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:40:51.366312 8669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:40:51.425406 8588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 4
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.62@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
I20260929 16:40:51.441898 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.426145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53166)
0929 16:40:51.426341 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.426361 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:51.426472 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:51.426818 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:51.426823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:51.427373 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:51.427603 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.427616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.429002 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.429007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.430468 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.430485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.431502 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.431508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.431743 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.432397 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.432441 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.435058 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.439040 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.439047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.439052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.439592 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.440005 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.440010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.440014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.440188 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.440892 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.440897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.441060 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.441202 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.441495 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:40:51.441928 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.426211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.17.205.60:34237)
0929 16:40:51.426431 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.426437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.426462 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:40:51.426571 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.426576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.426658 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.426778 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.427760 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.428878 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.430619 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.431345 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.431480 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.435279 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.435285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.435291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.435336 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.438896 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.439725 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.439732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.439735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.439904 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.440296 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.440301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.440304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.440777 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.440942 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.441317 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.441542 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:40:51.446470 8588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: Bootstrap starting.
I20260929 16:40:51.449878 8588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:51.451126 8588 log.cc:824] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: Log is configured to *not* fsync() on all Append() calls
I20260929 16:40:51.453961 8588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: No bootstrap required, opened a new log
I20260929 16:40:51.459256 8588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.459574 8588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c55e4b4f6f84ddc86212ba172578664, State: Initialized, Role: FOLLOWER
I20260929 16:40:51.460043 8588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.460964 8757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.461333 8757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.462109 8588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:51.462217 8563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:40:51.462425 8563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:51.466054 8563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.469936 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c55e4b4f6f84ddc86212ba172578664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e7a80bbc808499981a71e1d6590902f"
I20260929 16:40:51.470504 8711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:51.474311 8711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c55e4b4f6f84ddc86212ba172578664 in term 1.
I20260929 16:40:51.475044 8538 leader_election.cc:302] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7a80bbc808499981a71e1d6590902f, 9c55e4b4f6f84ddc86212ba172578664; no voters:
I20260929 16:40:51.475669 8757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:40:51.481115 8662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } attempt: 5
I20260929 16:40:51.482488 8563 leader_election.cc:288] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [CANDIDATE]: Term 1 election: Requested vote from peers e8c0f96111b1423ea3d243e6be17f186 (127.17.205.61:34527), 6e7a80bbc808499981a71e1d6590902f (127.17.205.60:34237)
Sep 29 16:40:51 dist-test-slave-3h01 krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700050, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.205.61@KRBTEST.COM for kudu/127.17.205.60@KRBTEST.COM
W20260929 16:40:51.498651 8690 tablet.cc:2369] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:51.472426 8521 master_runner.cc:186] Error getting master registration for 127.17.205.61:34527: OK, instance_id { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" instance_seqno: 1790700050761252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:40:51.500047 8521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:40:51.505007 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.481751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.205.60:34237 (local address 127.0.0.1:53170)
0929 16:40:51.482162 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 16:40:51.482182 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:40:51.482297 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:40:51.482729 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:40:51.482735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:40:51.483304 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:40:51.483525 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.483538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.485065 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.485071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.486487 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:40:51.486504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.487720 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.487727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:40:51.487964 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.489953 (+ 1989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:40:51.489977 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:40:51.493305 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:40:51.499797 (+ 6492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.499804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.499809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.500715 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.501990 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:40:51.501996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:40:51.502001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:40:51.502273 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:40:51.503397 (+ 1124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:40:51.503403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:40:51.503693 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 16:40:51.503890 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:40:51.504391 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260929 16:40:51.507157 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.482167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.17.205.60:34237)
0929 16:40:51.482373 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.482379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.482403 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:40:51.482491 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.482497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.482581 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.482690 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.483718 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.484886 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.486661 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.487389 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.487512 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.493665 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.493672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.493678 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.493721 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.499511 (+ 5790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.500997 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.501023 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.501030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.501796 (+ 766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.502506 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.502537 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.502541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.503159 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.505650 (+ 2491us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.505987 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.506466 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:40:51.513156 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: Bootstrap starting.
W20260929 16:40:51.513159 8768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:40:51.513375 8768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:40:51.513666 8757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [term 1 LEADER]: Becoming Leader. State: Replica: 9c55e4b4f6f84ddc86212ba172578664, State: Running, Role: LEADER
I20260929 16:40:51.514681 8757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.517362 8662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: Neither blocks nor log segments found. Creating new log.
I20260929 16:40:51.519377 8662 log.cc:824] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: Log is configured to *not* fsync() on all Append() calls
W20260929 16:40:51.526996 8542 tablet.cc:2369] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:40:51.542732 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: No bootstrap required, opened a new log
I20260929 16:40:51.547441 8770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55e4b4f6f84ddc86212ba172578664. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.547863 8770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:51.553874 8662 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.554334 8662 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c0f96111b1423ea3d243e6be17f186, State: Initialized, Role: FOLLOWER
I20260929 16:40:51.562951 8780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:40:51.565228 8780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:40:51.566226 8662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } }
I20260929 16:40:51.569991 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.527292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.17.205.61:34527)
0929 16:40:51.527479 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.527486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.530464 (+ 2978us) server_negotiation.cc:407] Connection header received
0929 16:40:51.530692 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.530698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.530776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.530846 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.536587 (+ 5741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.537596 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.543906 (+ 6310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.544584 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.544685 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.555508 (+ 10823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.555515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.555520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.555563 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.565456 (+ 9893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.566599 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.566605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.566608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.566760 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.567170 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.567174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.567178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.567599 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.567828 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.569340 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.569611 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:40:51.573975 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.518651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.17.205.60:34237)
0929 16:40:51.518861 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.518867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.526515 (+ 7648us) server_negotiation.cc:407] Connection header received
0929 16:40:51.526708 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.526714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.526833 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.526945 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.530924 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.532146 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.538652 (+ 6506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.539433 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.539542 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.554865 (+ 15323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.554873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.554878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.554929 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.563722 (+ 8793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.566282 (+ 2560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.566287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.566291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.566449 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.571916 (+ 5467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.571923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.571926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.572373 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.572626 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.573250 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.573542 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:40:51.575793 8781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.576264 8781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.578269 8637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c55e4b4f6f84ddc86212ba172578664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e8c0f96111b1423ea3d243e6be17f186"
I20260929 16:40:51.578722 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:51.517539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.17.205.62:45021)
0929 16:40:51.517757 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.517764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.526037 (+ 8273us) server_negotiation.cc:407] Connection header received
0929 16:40:51.527730 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.527736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.527822 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.527909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.532413 (+ 4504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.533463 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.536319 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:51.537052 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:51.537159 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:51.555764 (+ 18605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:40:51.555772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:40:51.555777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:40:51.555825 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:40:51.561901 (+ 6076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.567916 (+ 6015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.567926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.567930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.568087 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:40:51.576895 (+ 8808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:40:51.576902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:40:51.576906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:40:51.577322 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:40:51.577595 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:51.578128 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:40:51.578395 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:40:51.578713 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:40:51.581622 8637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c55e4b4f6f84ddc86212ba172578664 in term 1.
I20260929 16:40:51.582108 8662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:40:51.596305 8711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [term 1 FOLLOWER]: Refusing update from remote peer 9c55e4b4f6f84ddc86212ba172578664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:51.596611 8637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [term 1 FOLLOWER]: Refusing update from remote peer 9c55e4b4f6f84ddc86212ba172578664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:40:51.597292 8770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:51.597599 8757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:40:51.621381 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55e4b4f6f84ddc86212ba172578664. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.621692 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.632665 8788 mvcc.cc:205] Tried to move back new op lower bound from 7334707411328299008 to 7334707411014619136. Current Snapshot: MvccSnapshot[applied={T|T < 7334707411328299008}]
I20260929 16:40:51.640142 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.640435 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.648617 8770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.648974 8770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:51.650552 8780 catalog_manager.cc:1347] Generated new cluster ID: 1da1b8c3ecb7463c99278ca0d19e3dc5
I20260929 16:40:51.650678 8780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:40:51.669790 8781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55e4b4f6f84ddc86212ba172578664. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.670127 8781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.672593 8757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.672914 8757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664 [sys.catalog]: This master's current role is: LEADER
I20260929 16:40:51.673547 8789 mvcc.cc:205] Tried to move back new op lower bound from 7334707411328299008 to 7334707411014619136. Current Snapshot: MvccSnapshot[applied={T|T < 7334707411328299008}]
I20260929 16:40:51.688246 8781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c55e4b4f6f84ddc86212ba172578664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55e4b4f6f84ddc86212ba172578664" member_type: VOTER last_known_addr { host: "127.17.205.62" port: 45021 } } peers { permanent_uuid: "e8c0f96111b1423ea3d243e6be17f186" member_type: VOTER last_known_addr { host: "127.17.205.61" port: 34527 } } peers { permanent_uuid: "6e7a80bbc808499981a71e1d6590902f" member_type: VOTER last_known_addr { host: "127.17.205.60" port: 34237 } } }
I20260929 16:40:51.688540 8781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:40:51.707530 8801 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:40:51.711465 8780 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:40:51.713295 8780 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:40:51.739534 8780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c55e4b4f6f84ddc86212ba172578664: Generated new TSK 0
I20260929 16:40:51.740834 8780 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:40:51.840487 8741 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.827522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.17.205.62:45021)
0929 16:40:51.827708 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.827714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.838351 (+ 10637us) server_negotiation.cc:407] Connection header received
0929 16:40:51.838815 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.838822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.838912 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.838980 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.840205 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37840: BlockingRecv error: recv got EOF from 127.0.0.1:37840 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:40:51.841603 8739 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.827529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.17.205.60:34237)
0929 16:40:51.827713 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.827719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.839782 (+ 12063us) server_negotiation.cc:407] Connection header received
0929 16:40:51.840065 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.840070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.840150 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.840226 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.841348 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53198: BlockingRecv error: recv got EOF from 127.0.0.1:53198 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:40:51.842705 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.839944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.17.205.61:34527)
0929 16:40:51.840836 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.840843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.841214 (+ 371us) server_negotiation.cc:407] Connection header received
0929 16:40:51.841445 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.841451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.841538 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.841602 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.842505 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42982: BlockingRecv error: recv got EOF from 127.0.0.1:42982 (error 108)
Metrics: {"server-negotiator.queue_time_us":739}
W20260929 16:40:51.851605 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.849294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.17.205.61:34527)
0929 16:40:51.849477 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.849484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.849985 (+ 501us) server_negotiation.cc:407] Connection header received
0929 16:40:51.850181 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.850185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.850279 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.850365 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.851120 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42988: BlockingRecv error: recv got EOF from 127.0.0.1:42988 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:40:51.857685 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.855717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.17.205.61:34527)
0929 16:40:51.855912 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.855918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.856388 (+ 470us) server_negotiation.cc:407] Connection header received
0929 16:40:51.856534 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.856538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.856630 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.856722 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.857487 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42994: BlockingRecv error: recv got EOF from 127.0.0.1:42994 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 16:40:51.864656 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.862861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.17.205.61:34527)
0929 16:40:51.863055 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.863061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.863419 (+ 358us) server_negotiation.cc:407] Connection header received
0929 16:40:51.863558 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.863564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.863663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.863766 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.864419 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43002: BlockingRecv error: recv got EOF from 127.0.0.1:43002 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 16:40:51.873754 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.872079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.17.205.61:34527)
0929 16:40:51.872298 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.872305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.872541 (+ 236us) server_negotiation.cc:407] Connection header received
0929 16:40:51.872686 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.872692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.872790 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.872898 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.873544 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43008: BlockingRecv error: recv got EOF from 127.0.0.1:43008 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 16:40:51.887404 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.885255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.17.205.61:34527)
0929 16:40:51.885517 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.885524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.885816 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:40:51.886078 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.886083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.886159 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.886256 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.887122 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43010: BlockingRecv error: recv got EOF from 127.0.0.1:43010 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 16:40:51.909333 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.907243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.17.205.61:34527)
0929 16:40:51.907408 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.907414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.907751 (+ 337us) server_negotiation.cc:407] Connection header received
0929 16:40:51.907988 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.907993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.908070 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.908158 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.909119 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43026: BlockingRecv error: recv got EOF from 127.0.0.1:43026 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 16:40:51.947063 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:51.945063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.17.205.61:34527)
0929 16:40:51.945275 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:51.945282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:51.945629 (+ 347us) server_negotiation.cc:407] Connection header received
0929 16:40:51.945831 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:51.945835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:51.945917 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:51.946000 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:51.946791 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43040: BlockingRecv error: recv got EOF from 127.0.0.1:43040 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 16:40:52.016677 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:52.014756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.17.205.61:34527)
0929 16:40:52.014962 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.014968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.015360 (+ 392us) server_negotiation.cc:407] Connection header received
0929 16:40:52.015549 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.015555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.015645 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.015751 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:52.016461 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43044: BlockingRecv error: recv got EOF from 127.0.0.1:43044 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:40:52.150964 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:52.149244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.17.205.61:34527)
0929 16:40:52.149465 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.149471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.149702 (+ 231us) server_negotiation.cc:407] Connection header received
0929 16:40:52.149891 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.149895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.149965 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.150047 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:52.150726 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43054: BlockingRecv error: recv got EOF from 127.0.0.1:43054 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:40:52.359680 8754 catalog_manager.cc:1259] Loaded cluster ID: 1da1b8c3ecb7463c99278ca0d19e3dc5
I20260929 16:40:52.359789 8754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: loading cluster ID for follower catalog manager: success
I20260929 16:40:52.362361 8754 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: acquiring CA information for follower catalog manager: success
I20260929 16:40:52.364357 8754 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e7a80bbc808499981a71e1d6590902f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:40:52.414698 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:40:52.412972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.17.205.61:34527)
0929 16:40:52.413189 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.413195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.413319 (+ 124us) server_negotiation.cc:407] Connection header received
0929 16:40:52.413532 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.413537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.413624 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.413712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:40:52.414483 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43062: BlockingRecv error: recv got EOF from 127.0.0.1:43062 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:40:52.711720 8801 catalog_manager.cc:1259] Loaded cluster ID: 1da1b8c3ecb7463c99278ca0d19e3dc5
I20260929 16:40:52.711839 8801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: loading cluster ID for follower catalog manager: success
I20260929 16:40:52.714449 8801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: acquiring CA information for follower catalog manager: success
I20260929 16:40:52.716391 8801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8c0f96111b1423ea3d243e6be17f186: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:40:52.936677 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:52.930446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.17.205.61:34527)
0929 16:40:52.931162 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.931166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.931182 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:40:52.931282 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.931287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.931466 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.931574 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:52.932057 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.933039 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.934379 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.934812 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.934925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:52.935480 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:52.936037 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:52.936183 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:52.936285 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 16:40:52.936378 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":405,"threads_started":1}
I20260929 16:40:52.950050 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:52.943352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.17.205.60:34237)
0929 16:40:52.943880 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.943885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.944201 (+ 316us) server_negotiation.cc:407] Connection header received
0929 16:40:52.944417 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.944421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.944567 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.944701 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:52.945169 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.945861 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.947754 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.948179 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.948269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:52.948855 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:52.949427 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:52.949536 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:52.949663 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 16:40:52.949727 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260929 16:40:52.968937 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:52.957182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.17.205.60:34237)
0929 16:40:52.957424 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.957431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.960637 (+ 3206us) server_negotiation.cc:407] Connection header received
0929 16:40:52.960886 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.960891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.960986 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.961047 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:52.961983 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.963077 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.966356 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.967072 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.967182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:52.968003 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:52.968156 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:52.968238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:52.968479 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 16:40:52.968574 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:40:52.970230 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:52.956833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.17.205.62:45021)
0929 16:40:52.960313 (+ 3480us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.960320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.960339 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:40:52.960465 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.960472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.960688 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.961499 (+ 811us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:52.961700 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.962994 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.965440 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.966180 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.966307 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:52.967019 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:52.967516 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:52.967625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:52.969839 (+ 2214us) server_negotiation.cc:299] Negotiation successful
0929 16:40:52.969927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3334,"thread_start_us":3201,"threads_started":1}
I20260929 16:40:52.975091 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:40:52.960773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.17.205.61:34527)
0929 16:40:52.960971 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:40:52.960977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:40:52.968192 (+ 7215us) server_negotiation.cc:407] Connection header received
0929 16:40:52.968464 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:40:52.968470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:40:52.968555 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:40:52.968624 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:40:52.969395 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.970491 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.972376 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:40:52.972952 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:40:52.973095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:40:52.973659 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:40:52.973793 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:40:52.973894 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:40:52.974776 (+ 882us) server_negotiation.cc:299] Negotiation successful
0929 16:40:52.974861 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:40:52.975138 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8521
I20260929 16:40:52.976189 8583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:53.092839 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8521
W20260929 16:40:53.109750 8685 connection.cc:535] server connection from 127.0.0.1:53166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:40:53.110121 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8595
I20260929 16:40:53.111145 8657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:53.222556 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8595
I20260929 16:40:53.245994 18228 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu pid 8669
I20260929 16:40:53.247162 8731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:40:53.354014 18228 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaaWXO3/build/asan/bin/kudu with pid 8669
2026-09-29T16:40:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4744 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9573 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357404 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST